Jul 02 13:45:00.045711 ip-10-0-136-2 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 13:45:01.108139 ip-10-0-136-2 systemd[1]: Started Kubernetes Kubelet.
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.025469    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.030987    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.030998    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031001    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031010    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031014    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031017    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031020    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:45:01.144253 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031023    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031026    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031029    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031032    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031034    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031038    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031041    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031044    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031047    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031049    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031052    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031075    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031079    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031082    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031086    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031089    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031092    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031095    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031097    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:45:01.145295 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031100    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031103    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031105    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031110    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031114    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031117    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031120    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031123    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031127    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031130    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031134    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031137    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031139    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031142    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031145    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031149    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031152    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031154    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031158    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031161    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:45:01.146168 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031163    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031166    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031169    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031172    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031175    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031178    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031181    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031184    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031187    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031190    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031194    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031198    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031201    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031204    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031206    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031209    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031212    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031215    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031217    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031220    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:45:01.147069 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031223    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031225    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031228    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031231    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031234    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031236    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031240    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031243    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031246    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031248    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031251    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031254    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031257    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031260    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031263    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031265    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031268    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031271    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031274    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031276    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:45:01.147924 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031279    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031281    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031284    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031287    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031290    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031292    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031295    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031297    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031300    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031302    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031305    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031308    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031310    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.031999    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032006    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032009    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032013    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032016    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032020    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:45:01.148739 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032023    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032026    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032028    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032031    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032034    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032037    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032040    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032043    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032045    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032048    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032052    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032054    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032057    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032060    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032063    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032066    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032069    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032071    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032074    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032077    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:45:01.149588 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032080    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032083    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032085    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032088    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032091    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032094    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032097    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032100    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032102    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032105    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032108    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032111    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032115    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032119    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032124    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032127    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032130    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032133    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032136    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032138    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:45:01.150491 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032141    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032144    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032146    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032149    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032155    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032157    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032160    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032163    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032165    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032168    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032171    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032173    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032176    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032179    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032181    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032184    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032187    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032190    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032192    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:45:01.151382 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032196    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032199    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032201    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032204    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032207    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032210    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032213    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032215    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032219    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032221    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032224    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032228    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032232    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032236    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032240    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032242    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032245    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032250    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032253    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:45:01.152261 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032255    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032258    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032261    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032263    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032266    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032269    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032271    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032274    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032277    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032279    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032282    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032285    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032287    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032290    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032293    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032657    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032662    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032666    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032669    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032671    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:45:01.153120 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032676    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032680    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032683    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032686    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032689    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032692    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032695    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032698    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032700    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032703    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032709    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032711    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032714    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032717    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032719    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032722    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032725    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032728    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032730    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032733    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:45:01.153978 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032736    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032738    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032741    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032744    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032746    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032749    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032752    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032754    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032757    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032759    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032762    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032765    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032767    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032770    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032772    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032775    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032777    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032781    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032783    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032786    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:45:01.154858 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032789    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032791    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032795    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032798    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032801    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032803    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032806    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032809    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032812    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032814    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032817    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032819    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032822    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032824    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032827    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032830    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032832    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032835    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032838    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:45:01.156182 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032840    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032843    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032846    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032848    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032852    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032856    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032860    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032863    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032866    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032869    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032872    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032874    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032877    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032881    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032887    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032891    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032894    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032897    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032899    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:45:01.157045 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032902    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032905    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032907    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032910    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032913    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032923    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032928    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032931    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032933    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032936    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032939    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032941    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032944    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032946    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032949    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.032952    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033044    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033050    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033057    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033063    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033069    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 13:45:01.157938 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033075    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033081    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033087    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033092    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033095    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033099    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033104    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033109    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033112    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033115    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033119    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033122    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033125    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033128    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033131    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033135    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033139    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033142    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033146    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033149    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033152    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033156    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033159    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033163    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 13:45:01.158811 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033166    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033169    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033173    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033177    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033181    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033184    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033195    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033199    2566 flags.go:64] FLAG: --enable-server="true"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033202    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033206    2566 flags.go:64] FLAG: --event-burst="100"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033210    2566 flags.go:64] FLAG: --event-qps="50"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033213    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033216    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033219    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033224    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033229    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033232    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033235    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033239    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033242    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033245    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033248    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033251    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033254    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033257    2566 flags.go:64] FLAG: --feature-gates=""
Jul 02 13:45:01.159701 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033261    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033264    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033268    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033271    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033274    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033278    2566 flags.go:64] FLAG: --help="false"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033281    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-136-2.ec2.internal"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033284    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033287    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033290    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033294    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033297    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033300    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033303    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033698    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033702    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033705    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033709    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033712    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033715    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033718    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033723    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033727    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033730    2566 flags.go:64] FLAG: --lock-file=""
Jul 02 13:45:01.160988 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033734    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033737    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033740    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033745    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033748    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033752    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033755    2566 flags.go:64] FLAG: --logging-format="text"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033758    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033762    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033765    2566 flags.go:64] FLAG: --manifest-url=""
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033768    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033772    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033775    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033779    2566 flags.go:64] FLAG: --max-pods="110"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033782    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033785    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033788    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033792    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033795    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033798    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033801    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033808    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033811    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033815    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 13:45:01.165410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033818    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033821    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033827    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033830    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033834    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033837    2566 flags.go:64] FLAG: --port="10250"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033843    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033846    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0901df2661074c142"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033850    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033853    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033857    2566 flags.go:64] FLAG: --register-node="true"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033860    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033863    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033866    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033869    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033872    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033876    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033879    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033883    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033886    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033889    2566 flags.go:64] FLAG: --runonce="false"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033892    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033895    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033898    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033901    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 13:45:01.166272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033904    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033907    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033910    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033914    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033917    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033920    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033923    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033926    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033929    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033932    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033936    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033941    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033944    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033950    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033953    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033956    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033959    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033962    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033966    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033969    2566 flags.go:64] FLAG: --v="2"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033973    2566 flags.go:64] FLAG: --version="false"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033977    2566 flags.go:64] FLAG: --vmodule=""
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033981    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.033984    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034095    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:45:01.168490 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034099    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034102    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034105    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034108    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034112    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034116    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034120    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034123    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034126    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034129    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034132    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034135    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034138    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034140    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034143    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034147    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034149    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034152    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034156    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:45:01.169391 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034160    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034167    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034170    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034173    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034176    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034178    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034181    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034184    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034187    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034190    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034193    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034196    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034199    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034201    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034204    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034207    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034210    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034212    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034215    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034218    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:45:01.170695 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034221    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034223    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034226    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034229    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034232    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034248    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034251    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034254    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034257    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034260    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034262    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034265    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034268    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034273    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034276    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034279    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034282    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034284    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034287    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034290    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:45:01.171575 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034292    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034295    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034298    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034301    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034304    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034307    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034309    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034312    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034315    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034317    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034320    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034322    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034325    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034327    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034330    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034333    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034336    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034338    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034341    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034344    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:45:01.286613 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034347    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034349    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034352    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034355    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034358    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034363    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034366    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034368    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034371    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034373    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034376    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034379    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034382    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034384    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034387    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034390    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034393    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034395    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:45:01.287435 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.034399    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.035113    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.046201    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.046217    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.046238    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.046244    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046302    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046307    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046311    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046314    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046317    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046321    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046324    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046327    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046330    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:45:01.289189 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046333    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046336    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046340    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046343    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046346    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046349    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046352    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046355    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046358    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046361    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046364    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046367    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046370    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046372    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046376    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046378    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046381    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046384    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046387    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046390    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:45:01.290097 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046393    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046396    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046401    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046405    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046408    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046412    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046415    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046418    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046421    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046424    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046427    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046431    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046434    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046437    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046440    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046443    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046446    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046449    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046452    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:45:01.291375 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046455    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046458    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046460    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046463    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046466    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046469    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046471    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046474    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046477    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046480    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046483    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046486    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046488    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046491    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046494    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046497    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046499    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046502    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046505    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046507    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:45:01.298887 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046514    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046517    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046519    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046522    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046526    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046531    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046534    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046541    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046543    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046546    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046549    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046575    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046581    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046585    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046588    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046591    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046594    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046597    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046600    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:45:01.299749 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046603    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046605    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046608    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046611    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046614    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046617    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046620    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046623    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046626    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046629    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046631    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046634    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.046640    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046753    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046760    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:45:01.300596 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046763    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046766    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046769    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046771    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046774    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046778    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046781    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046784    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046787    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046790    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046793    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046796    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046799    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046801    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046804    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046807    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046810    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046813    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046815    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046818    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:45:01.303222 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046821    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046825    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046830    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046833    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046836    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046839    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046842    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046844    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046847    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046850    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046852    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046857    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046860    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046862    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046865    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046868    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046871    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046875    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046878    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:45:01.304113 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046880    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046883    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046886    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046888    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046892    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046896    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046899    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046903    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046906    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046909    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046912    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046915    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046917    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046921    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046925    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046928    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046931    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046934    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046937    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:45:01.305003 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046940    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046943    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046946    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046949    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046951    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046955    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046958    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046961    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046964    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046966    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046969    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046973    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046976    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046979    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046982    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046984    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046987    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046990    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046994    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046996    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:45:01.308526 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.046999    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047002    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047005    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047007    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047010    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047013    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047016    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047019    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047022    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047024    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047027    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047030    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047033    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047036    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047038    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047041    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047044    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047048    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:45:01.309385 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:01.047050    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.047056    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.047150    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.049535    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.051225    2566 server.go:1019] "Starting client certificate rotation"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.051332    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.051373    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.053492    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.055854    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.073580    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.076787    2566 log.go:25] "Validated CRI v1 image API"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.077705    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 13:45:01.418011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.080363    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9728126f-5aff-4406-810e-59bf41ddc99a:/dev/nvme0n1p4 bdaa0167-fa08-49fc-827e-e6d3e5162f9b:/dev/nvme0n1p3]
Jul 02 13:45:01.419259 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.080376    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 13:45:01.421027 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.085848    2566 manager.go:216] Machine: {Timestamp:2026-07-02 13:45:01.084912779 +0000 UTC m=+0.926550288 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199644 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2632244585c9ace99abe9095481933 SystemUUID:ec263224-4585-c9ac-e99a-be9095481933 BootID:b2a3bb3e-3b0c-489f-8bf3-b82863f9ea6a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:81:f4:ac:7d:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:81:f4:ac:7d:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:f4:6c:95:d1:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 13:45:01.421027 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.085949    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 13:45:01.421027 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.086040    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 13:45:01.421027 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.087543    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 13:45:01.421027 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.087583    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-2.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 13:45:01.421027 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.089036    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 13:45:01.421027 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.089051    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 13:45:01.421027 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.089061    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.089823    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.091087    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.094956    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.095089    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.096015    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.096029    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.096047    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.096063    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.096074    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.096407    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.096427    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.098649    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.100297    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.100325    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101015    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101033    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101039    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101045    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101054    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101060    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101224    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101229    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101235    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101241    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101249    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.101258    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.102935    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.102943    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.105760    2566 server.go:1268] "Started kubelet"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.105827    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.105912    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.105987    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.108310    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.109344    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.112286    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.117627    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.117658    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.117867    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118390    2566 factory.go:55] Registering systemd factory
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118402    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118598    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118600    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118629    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118637    2566 factory.go:153] Registering CRI-O factory
Jul 02 13:45:01.421448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118651    2566 factory.go:223] Registration of the crio container factory successfully
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118705    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118710    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118720    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118733    2566 factory.go:103] Registering Raw factory
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.118748    2566 manager.go:1196] Started watching for new ooms in manager
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.119179    2566 manager.go:349] Starting recovery of all containers
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.131280    2566 manager.go:354] Recovery completed
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:01.135425    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.218928    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.319428    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:01.423199 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.419954    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:01.545036 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.520600    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:01.673512 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.621531    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:01.804501 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.721840    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:01.826996 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.822399    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:01.940290 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:01.922831    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.078064 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.023404    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.211405 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.124538    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.332544 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.224829    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.332544 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.325348    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.425858 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.425846    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.451432 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.451411    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:45:02.451530 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.451451    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:45:02.451530 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.451462    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:45:02.452086 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.452063    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 13:45:02.452160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.452087    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 13:45:02.452160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.452111    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:45:02.452160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.452069    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:45:02.453281 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.453268    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:45:02.453330 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.453294    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:45:02.453330 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.453311    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:45:02.456916 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.456889    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:45:02.457028 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.456936    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k5hwc"
Jul 02 13:45:02.457713 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.457699    2566 policy_none.go:49] "None policy: Start"
Jul 02 13:45:02.457771 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.457716    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 13:45:02.457771 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.457726    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 13:45:02.458798 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.458780    2566 policy_none.go:47] "Start"
Jul 02 13:45:02.461644 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.461616    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 13:45:02.461735 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.461653    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-2.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 13:45:02.461805 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.461746    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 13:45:02.461805 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.461796    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 13:45:02.461901 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.461839    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 13:45:02.462667 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.461707    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-2.ec2.internal.18be7cf69bf6c302  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-2.ec2.internal,UID:ip-10-0-136-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-2.ec2.internal,},FirstTimestamp:2026-07-02 13:45:01.105734402 +0000 UTC m=+0.947371909,LastTimestamp:2026-07-02 13:45:01.105734402 +0000 UTC m=+0.947371909,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-2.ec2.internal,}"
Jul 02 13:45:02.465308 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.465248    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-2.ec2.internal.18be7cf6ec2c80c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-2.ec2.internal,UID:ip-10-0-136-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-2.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-2.ec2.internal,},FirstTimestamp:2026-07-02 13:45:02.451433673 +0000 UTC m=+2.293071184,LastTimestamp:2026-07-02 13:45:02.451433673 +0000 UTC m=+2.293071184,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-2.ec2.internal,}"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.496189    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.497476    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.497507    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.497524    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.497552    2566 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.498394    2566 manager.go:337] "Starting Device Plugin manager"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.498427    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.498441    2566 server.go:87] "Starting device plugin registration server"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.498704    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.498719    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.498920    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.499001    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.499013    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.499498    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.499550    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.501081 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.500101    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:45:02.599198 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.599141    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:45:02.599198 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.599160    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal"]
Jul 02 13:45:02.599312 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.599243    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:45:02.599978 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.599960    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:45:02.600055 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.599991    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:45:02.600055 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.600002    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:45:02.600055 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.600025    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.600675 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.600647    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:45:02.600728 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.600689    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:45:02.600728 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.600703    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:45:02.602397 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.602373    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:45:02.602616 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.602601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.602709 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.602646    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:45:02.603516 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.603497    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:45:02.603616 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.603527    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:45:02.603616 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.603542    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:45:02.603731 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.603703    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:45:02.603773 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.603745    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:45:02.603773 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.603761    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:45:02.604908 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.604893    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.604970 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.604918    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:45:02.605670 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.605656    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:45:02.605745 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.605684    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:45:02.605745 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.605697    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:45:02.609505 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.609487    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.609633 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.609514    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-2.ec2.internal\": node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.626071 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.626052    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.627224 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.627207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24dda35e4ab82f2be5688279542e5a7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal\" (UID: \"24dda35e4ab82f2be5688279542e5a7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.627293 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.627234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24dda35e4ab82f2be5688279542e5a7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal\" (UID: \"24dda35e4ab82f2be5688279542e5a7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.627293 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.627249    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-2.ec2.internal\" not found" node="ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.627293 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.627252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99227ef48a198d127b2226d0b66cb13b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-2.ec2.internal\" (UID: \"99227ef48a198d127b2226d0b66cb13b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.632027 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.632012    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-2.ec2.internal\" not found" node="ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.726344 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.726320    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.727539 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.727516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24dda35e4ab82f2be5688279542e5a7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal\" (UID: \"24dda35e4ab82f2be5688279542e5a7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.727655 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.727544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24dda35e4ab82f2be5688279542e5a7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal\" (UID: \"24dda35e4ab82f2be5688279542e5a7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.727655 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.727563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99227ef48a198d127b2226d0b66cb13b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-2.ec2.internal\" (UID: \"99227ef48a198d127b2226d0b66cb13b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.727655 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.727611    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99227ef48a198d127b2226d0b66cb13b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-2.ec2.internal\" (UID: \"99227ef48a198d127b2226d0b66cb13b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.727655 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.727633    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24dda35e4ab82f2be5688279542e5a7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal\" (UID: \"24dda35e4ab82f2be5688279542e5a7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.727655 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.727642    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24dda35e4ab82f2be5688279542e5a7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal\" (UID: \"24dda35e4ab82f2be5688279542e5a7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.826846 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.826815    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.928021 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:02.927968    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:02.931454 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.931435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:02.936697 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:02.936679    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:03.028373 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:03.028346    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:03.052283 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.052263    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 13:45:03.052433 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.052410    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 13:45:03.128565 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:03.128536    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:03.229113 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:03.229056    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:03.329268 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:03.329239    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:03.430325 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:03.430295    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:03.465838 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.465809    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 13:40:01 +0000 UTC" deadline="2028-03-03 23:30:44.405306321 +0000 UTC"
Jul 02 13:45:03.465838 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.465834    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14649h45m40.939474952s"
Jul 02 13:45:03.508110 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.508050    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:45:03.530726 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:03.530702    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:03.630359 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.630336    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:45:03.630824 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:03.630805    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:03.631459 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.631439    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:45:03.631542 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.631474    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:45:03.631542 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.631490    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:45:03.731595 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:03.731561    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-2.ec2.internal\" not found"
Jul 02 13:45:03.808384 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.808366    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:45:03.818640 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.818612    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:03.828161 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.828135    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:45:03.831417 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.831401    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:45:03.832570 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.832544    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal"
Jul 02 13:45:03.846121 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.846101    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:45:03.874434 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:03.874405    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dda35e4ab82f2be5688279542e5a7f.slice/crio-5bff47e895f82b8b6f16201676a1c38c7bf8081b72c6814068bc1090c2c12656 WatchSource:0}: Error finding container 5bff47e895f82b8b6f16201676a1c38c7bf8081b72c6814068bc1090c2c12656: Status 404 returned error can't find the container with id 5bff47e895f82b8b6f16201676a1c38c7bf8081b72c6814068bc1090c2c12656
Jul 02 13:45:03.880168 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:03.880154    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:45:04.101201 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.101144    2566 apiserver.go:52] "Watching apiserver"
Jul 02 13:45:04.108954 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.108929    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 13:45:04.110139 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.110106    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-rq799","openshift-image-registry/node-ca-tnjj7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal","openshift-multus/multus-94mkh","openshift-multus/multus-additional-cni-plugins-4hzt5","openshift-multus/network-metrics-daemon-dm4g2","kube-system/global-pull-secret-syncer-5mxmz","kube-system/konnectivity-agent-wh4zs","kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal","openshift-network-diagnostics/network-check-target-zn2d9","openshift-network-operator/iptables-alerter-6nmmk","openshift-ovn-kubernetes/ovnkube-node-j86jl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7","openshift-cluster-node-tuning-operator/tuned-fpdcn"]
Jul 02 13:45:04.112331 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.112314    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.113575 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.113537    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.115452 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.115316    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.115591 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.115466    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.115591 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.115473    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.116322 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.116290    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.116779 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.116755    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8cnhf\""
Jul 02 13:45:04.117920 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.117894    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:04.118278 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.118127    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 13:45:04.118278 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.118196    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.118278 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.118262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wrjv5\""
Jul 02 13:45:04.119338 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.119114    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 13:45:04.119451 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.119367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.120391 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.120369    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.120883 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.120868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:04.121028 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.120986    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:04.122022 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.121995    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ghqt8\""
Jul 02 13:45:04.122022 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122010    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q5wbr\""
Jul 02 13:45:04.122201 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122061    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.122201 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122089    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 13:45:04.122437 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122413    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.122576 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122541    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 13:45:04.122576 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122550    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.122748 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122691    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 13:45:04.122748 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.122697    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:04.122748 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122695    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 13:45:04.122908 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122757    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 13:45:04.122908 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.122855    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-865zt\""
Jul 02 13:45:04.123247 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.123228    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.124190 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.124172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.125510 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.125474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:04.125616 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.125556    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:04.126590 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.126570    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 13:45:04.126695 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.126660    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sf6kw\""
Jul 02 13:45:04.126984 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.126945    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.128369 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.128346    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.129648 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.129623    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.129648 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.129638    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.130019 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.129942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.130278 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.130118    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ststz\""
Jul 02 13:45:04.130359 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.130325    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 13:45:04.131546 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.131525    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.131684 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.131621    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 13:45:04.131744 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.131675    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 13:45:04.131970 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.131952    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 13:45:04.132056 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.132036    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.132129 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.132109    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 13:45:04.132208 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.132189    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2xrjc\""
Jul 02 13:45:04.132412 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.132387    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 13:45:04.132484 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.132417    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.132632 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.132616    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k8dg6\""
Jul 02 13:45:04.134274 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-kubernetes\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134322 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.134366 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-cnibin\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.134405 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-sysconfig\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134444 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.134498 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a804050-c3a7-4cf3-923b-5146f3dde7c3-hosts-file\") pod \"node-resolver-rq799\" (UID: \"2a804050-c3a7-4cf3-923b-5146f3dde7c3\") " pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.134552 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.134552 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mbm\" (UniqueName: \"kubernetes.io/projected/2a804050-c3a7-4cf3-923b-5146f3dde7c3-kube-api-access-t9mbm\") pod \"node-resolver-rq799\" (UID: \"2a804050-c3a7-4cf3-923b-5146f3dde7c3\") " pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.134674 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/095b3c78-4147-4098-b517-722b1e4941f1-host\") pod \"node-ca-tnjj7\" (UID: \"095b3c78-4147-4098-b517-722b1e4941f1\") " pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.134674 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-var-lib-kubelet\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134674 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134609    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/095b3c78-4147-4098-b517-722b1e4941f1-serviceca\") pod \"node-ca-tnjj7\" (UID: \"095b3c78-4147-4098-b517-722b1e4941f1\") " pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.134674 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-systemd\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134674 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31be0683-8d85-4b98-927f-7161f9326721-tmp\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134674 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-system-cni-dir\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134697    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-os-release\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvvs\" (UniqueName: \"kubernetes.io/projected/095b3c78-4147-4098-b517-722b1e4941f1-kube-api-access-jqvvs\") pod \"node-ca-tnjj7\" (UID: \"095b3c78-4147-4098-b517-722b1e4941f1\") " pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-modprobe-d\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-sysctl-conf\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-run\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-sys\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjfv\" (UniqueName: \"kubernetes.io/projected/31be0683-8d85-4b98-927f-7161f9326721-kube-api-access-hzjfv\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ca65baf-3992-41ea-b4b9-e24681de33d3-agent-certs\") pod \"konnectivity-agent-wh4zs\" (UID: \"0ca65baf-3992-41ea-b4b9-e24681de33d3\") " pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ca65baf-3992-41ea-b4b9-e24681de33d3-konnectivity-ca\") pod \"konnectivity-agent-wh4zs\" (UID: \"0ca65baf-3992-41ea-b4b9-e24681de33d3\") " pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:04.134944 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-sysctl-d\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.135481 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-lib-modules\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.135481 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.134993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-host\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.135481 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.135010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31be0683-8d85-4b98-927f-7161f9326721-etc-tuned\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.135481 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.135026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8fq\" (UniqueName: \"kubernetes.io/projected/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-kube-api-access-2j8fq\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.135481 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.135045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a804050-c3a7-4cf3-923b-5146f3dde7c3-tmp-dir\") pod \"node-resolver-rq799\" (UID: \"2a804050-c3a7-4cf3-923b-5146f3dde7c3\") " pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.135857 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.135833    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.138461 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.138440    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:45:04.165344 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.165310    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mblfn"
Jul 02 13:45:04.178809 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.178792    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mblfn"
Jul 02 13:45:04.219448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.219424    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 13:45:04.224213 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.224196    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99227ef48a198d127b2226d0b66cb13b.slice/crio-fb305675cbb0354a71c250c77586344c350cde081c0c3ea2bc1b9d2679652f0c WatchSource:0}: Error finding container fb305675cbb0354a71c250c77586344c350cde081c0c3ea2bc1b9d2679652f0c: Status 404 returned error can't find the container with id fb305675cbb0354a71c250c77586344c350cde081c0c3ea2bc1b9d2679652f0c
Jul 02 13:45:04.235576 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/095b3c78-4147-4098-b517-722b1e4941f1-serviceca\") pod \"node-ca-tnjj7\" (UID: \"095b3c78-4147-4098-b517-722b1e4941f1\") " pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-systemd\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-os-release\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-cnibin\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-var-lib-cni-multus\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-run\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-sys\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235719    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-systemd\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjfv\" (UniqueName: \"kubernetes.io/projected/31be0683-8d85-4b98-927f-7161f9326721-kube-api-access-hzjfv\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235693    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-os-release\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ca65baf-3992-41ea-b4b9-e24681de33d3-konnectivity-ca\") pod \"konnectivity-agent-wh4zs\" (UID: \"0ca65baf-3992-41ea-b4b9-e24681de33d3\") " pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235770    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-sys\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-run-netns\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-var-lib-openvswitch\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.236101 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-run-ovn-kubernetes\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235843    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-run\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-cni-bin\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-sysctl-d\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-lib-modules\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235975    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-sysctl-d\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.235981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-host\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236008    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-host\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31be0683-8d85-4b98-927f-7161f9326721-etc-tuned\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236041    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/095b3c78-4147-4098-b517-722b1e4941f1-serviceca\") pod \"node-ca-tnjj7\" (UID: \"095b3c78-4147-4098-b517-722b1e4941f1\") " pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-run-systemd\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236071    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-lib-modules\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-log-socket\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a804050-c3a7-4cf3-923b-5146f3dde7c3-tmp-dir\") pod \"node-resolver-rq799\" (UID: \"2a804050-c3a7-4cf3-923b-5146f3dde7c3\") " pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-slash\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.236763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a8440fa-890d-46a8-ae75-c0c25838ebe4-ovn-node-metrics-cert\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-dbus\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-device-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236230    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-var-lib-cni-bin\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236278    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ca65baf-3992-41ea-b4b9-e24681de33d3-konnectivity-ca\") pod \"konnectivity-agent-wh4zs\" (UID: \"0ca65baf-3992-41ea-b4b9-e24681de33d3\") " pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-registration-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvl2r\" (UniqueName: \"kubernetes.io/projected/182a7a63-1fd0-452c-800f-e8866add61bf-kube-api-access-xvl2r\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-multus-cni-dir\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236362    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a804050-c3a7-4cf3-923b-5146f3dde7c3-tmp-dir\") pod \"node-resolver-rq799\" (UID: \"2a804050-c3a7-4cf3-923b-5146f3dde7c3\") " pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-run-netns\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.237425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-multus-conf-dir\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-etc-kubernetes\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47c5\" (UniqueName: \"kubernetes.io/projected/a40daca0-6d80-4669-b2de-895ea47b7087-kube-api-access-j47c5\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-cnibin\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-sysconfig\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236612    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-cnibin\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236686    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-sysconfig\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236745    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-run-openvswitch\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-kubelet-config\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-socket-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-run-multus-certs\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.238111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236879    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a780d76b-930b-46f0-96c7-ffab4bc1166a-host-slash\") pod \"iptables-alerter-6nmmk\" (UID: \"a780d76b-930b-46f0-96c7-ffab4bc1166a\") " pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sjnq\" (UniqueName: \"kubernetes.io/projected/a780d76b-930b-46f0-96c7-ffab4bc1166a-kube-api-access-5sjnq\") pod \"iptables-alerter-6nmmk\" (UID: \"a780d76b-930b-46f0-96c7-ffab4bc1166a\") " pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-systemd-units\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.236994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a8440fa-890d-46a8-ae75-c0c25838ebe4-ovnkube-script-lib\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237019    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-system-cni-dir\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-multus-socket-dir-parent\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-hostroot\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31be0683-8d85-4b98-927f-7161f9326721-tmp\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-system-cni-dir\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a780d76b-930b-46f0-96c7-ffab4bc1166a-iptables-alerter-script\") pod \"iptables-alerter-6nmmk\" (UID: \"a780d76b-930b-46f0-96c7-ffab4bc1166a\") " pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvvs\" (UniqueName: \"kubernetes.io/projected/095b3c78-4147-4098-b517-722b1e4941f1-kube-api-access-jqvvs\") pod \"node-ca-tnjj7\" (UID: \"095b3c78-4147-4098-b517-722b1e4941f1\") " pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237214    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-system-cni-dir\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.238669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-modprobe-d\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237276    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-sysctl-conf\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ca65baf-3992-41ea-b4b9-e24681de33d3-agent-certs\") pod \"konnectivity-agent-wh4zs\" (UID: \"0ca65baf-3992-41ea-b4b9-e24681de33d3\") " pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a40daca0-6d80-4669-b2de-895ea47b7087-cni-binary-copy\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8fq\" (UniqueName: \"kubernetes.io/projected/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-kube-api-access-2j8fq\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237384    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-modprobe-d\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-sys-fs\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-var-lib-kubelet\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-run-k8s-cni-cncf-io\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-kubernetes\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-node-log\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7qc\" (UniqueName: \"kubernetes.io/projected/66cf39f9-4496-49ad-aabc-e9453dab385a-kube-api-access-2b7qc\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237707    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.239135 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-os-release\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237707    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-sysctl-conf\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-kubelet\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237802    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-etc-kubernetes\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-run-ovn\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-cni-netd\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2qc\" (UniqueName: \"kubernetes.io/projected/5a8440fa-890d-46a8-ae75-c0c25838ebe4-kube-api-access-4p2qc\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a804050-c3a7-4cf3-923b-5146f3dde7c3-hosts-file\") pod \"node-resolver-rq799\" (UID: \"2a804050-c3a7-4cf3-923b-5146f3dde7c3\") " pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-etc-openvswitch\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a8440fa-890d-46a8-ae75-c0c25838ebe4-ovnkube-config\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a40daca0-6d80-4669-b2de-895ea47b7087-multus-daemon-config\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mbm\" (UniqueName: \"kubernetes.io/projected/2a804050-c3a7-4cf3-923b-5146f3dde7c3-kube-api-access-t9mbm\") pod \"node-resolver-rq799\" (UID: \"2a804050-c3a7-4cf3-923b-5146f3dde7c3\") " pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.239691 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/095b3c78-4147-4098-b517-722b1e4941f1-host\") pod \"node-ca-tnjj7\" (UID: \"095b3c78-4147-4098-b517-722b1e4941f1\") " pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-var-lib-kubelet\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238138    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238143    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a8440fa-890d-46a8-ae75-c0c25838ebe4-env-overrides\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238252    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31be0683-8d85-4b98-927f-7161f9326721-var-lib-kubelet\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238409    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/095b3c78-4147-4098-b517-722b1e4941f1-host\") pod \"node-ca-tnjj7\" (UID: \"095b3c78-4147-4098-b517-722b1e4941f1\") " pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.237991    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a804050-c3a7-4cf3-923b-5146f3dde7c3-hosts-file\") pod \"node-resolver-rq799\" (UID: \"2a804050-c3a7-4cf3-923b-5146f3dde7c3\") " pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.238728    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.239735    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31be0683-8d85-4b98-927f-7161f9326721-etc-tuned\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.239823    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31be0683-8d85-4b98-927f-7161f9326721-tmp\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.240122 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.240022    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ca65baf-3992-41ea-b4b9-e24681de33d3-agent-certs\") pod \"konnectivity-agent-wh4zs\" (UID: \"0ca65baf-3992-41ea-b4b9-e24681de33d3\") " pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:04.249129 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.249108    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjfv\" (UniqueName: \"kubernetes.io/projected/31be0683-8d85-4b98-927f-7161f9326721-kube-api-access-hzjfv\") pod \"tuned-fpdcn\" (UID: \"31be0683-8d85-4b98-927f-7161f9326721\") " pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.250015 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.249996    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8fq\" (UniqueName: \"kubernetes.io/projected/04a7dc11-0af2-4222-b6bf-06c30c9f50c0-kube-api-access-2j8fq\") pod \"multus-additional-cni-plugins-4hzt5\" (UID: \"04a7dc11-0af2-4222-b6bf-06c30c9f50c0\") " pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.251379 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.251359    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mbm\" (UniqueName: \"kubernetes.io/projected/2a804050-c3a7-4cf3-923b-5146f3dde7c3-kube-api-access-t9mbm\") pod \"node-resolver-rq799\" (UID: \"2a804050-c3a7-4cf3-923b-5146f3dde7c3\") " pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.251552 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.251533    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvvs\" (UniqueName: \"kubernetes.io/projected/095b3c78-4147-4098-b517-722b1e4941f1-kube-api-access-jqvvs\") pod \"node-ca-tnjj7\" (UID: \"095b3c78-4147-4098-b517-722b1e4941f1\") " pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.338940 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.338921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-run-k8s-cni-cncf-io\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.339056 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.338955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-node-log\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339056 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339002    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-node-log\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339056 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339008    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-run-k8s-cni-cncf-io\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.339056 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7qc\" (UniqueName: \"kubernetes.io/projected/66cf39f9-4496-49ad-aabc-e9453dab385a-kube-api-access-2b7qc\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-os-release\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339101    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-kubelet\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-run-ovn\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.339130    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-cni-netd\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2qc\" (UniqueName: \"kubernetes.io/projected/5a8440fa-890d-46a8-ae75-c0c25838ebe4-kube-api-access-4p2qc\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339191    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-kubelet\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.339210    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs podName:66cf39f9-4496-49ad-aabc-e9453dab385a nodeName:}" failed. No retries permitted until 2026-07-02 13:45:04.839169748 +0000 UTC m=+4.680807242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs") pod "network-metrics-daemon-dm4g2" (UID: "66cf39f9-4496-49ad-aabc-e9453dab385a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339223    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-os-release\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339237    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-run-ovn\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339243    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-cni-netd\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-etc-openvswitch\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a8440fa-890d-46a8-ae75-c0c25838ebe4-ovnkube-config\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a40daca0-6d80-4669-b2de-895ea47b7087-multus-daemon-config\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a8440fa-890d-46a8-ae75-c0c25838ebe4-env-overrides\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339407    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-etc-openvswitch\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-cnibin\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-var-lib-cni-multus\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-run-netns\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-var-lib-openvswitch\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339538    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-var-lib-cni-multus\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-run-ovn-kubernetes\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-cni-bin\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339579    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-cnibin\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-run-systemd\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.339957 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-log-socket\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-slash\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a8440fa-890d-46a8-ae75-c0c25838ebe4-ovn-node-metrics-cert\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-dbus\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-device-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339747    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-run-systemd\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339777    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-var-lib-openvswitch\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-var-lib-cni-bin\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339831    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-log-socket\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339841    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-cni-bin\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-registration-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339881    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-slash\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.340689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvl2r\" (UniqueName: \"kubernetes.io/projected/182a7a63-1fd0-452c-800f-e8866add61bf-kube-api-access-xvl2r\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339913    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a8440fa-890d-46a8-ae75-c0c25838ebe4-env-overrides\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-multus-cni-dir\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339942    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-run-netns\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339981    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a8440fa-890d-46a8-ae75-c0c25838ebe4-ovnkube-config\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339986    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-var-lib-cni-bin\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.339995    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-run-netns\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-multus-conf-dir\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340039    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-registration-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340042    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-dbus\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340041    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-device-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-etc-kubernetes\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340082    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-multus-conf-dir\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340041    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a40daca0-6d80-4669-b2de-895ea47b7087-multus-daemon-config\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340085    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-multus-cni-dir\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47c5\" (UniqueName: \"kubernetes.io/projected/a40daca0-6d80-4669-b2de-895ea47b7087-kube-api-access-j47c5\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340113    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-run-ovn-kubernetes\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340124    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-etc-kubernetes\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340125    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340180    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-host-run-netns\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-run-openvswitch\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-kubelet-config\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340244    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-run-openvswitch\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-socket-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.340254    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-run-multus-certs\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340296    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-kubelet-config\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.341730 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.340316    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret podName:4ad7ad64-df1a-46f9-8e19-2b7333b838f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:04.840299974 +0000 UTC m=+4.681937482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret") pod "global-pull-secret-syncer-5mxmz" (UID: "4ad7ad64-df1a-46f9-8e19-2b7333b838f5") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340349    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a780d76b-930b-46f0-96c7-ffab4bc1166a-host-slash\") pod \"iptables-alerter-6nmmk\" (UID: \"a780d76b-930b-46f0-96c7-ffab4bc1166a\") " pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340355    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-run-multus-certs\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340379    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-socket-dir\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sjnq\" (UniqueName: \"kubernetes.io/projected/a780d76b-930b-46f0-96c7-ffab4bc1166a-kube-api-access-5sjnq\") pod \"iptables-alerter-6nmmk\" (UID: \"a780d76b-930b-46f0-96c7-ffab4bc1166a\") " pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340419    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a780d76b-930b-46f0-96c7-ffab4bc1166a-host-slash\") pod \"iptables-alerter-6nmmk\" (UID: \"a780d76b-930b-46f0-96c7-ffab4bc1166a\") " pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-systemd-units\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a8440fa-890d-46a8-ae75-c0c25838ebe4-ovnkube-script-lib\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340461    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a8440fa-890d-46a8-ae75-c0c25838ebe4-systemd-units\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-system-cni-dir\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-multus-socket-dir-parent\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340543    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-system-cni-dir\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342146 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-hostroot\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340594    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-multus-socket-dir-parent\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a780d76b-930b-46f0-96c7-ffab4bc1166a-iptables-alerter-script\") pod \"iptables-alerter-6nmmk\" (UID: \"a780d76b-930b-46f0-96c7-ffab4bc1166a\") " pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340627    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-hostroot\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a40daca0-6d80-4669-b2de-895ea47b7087-cni-binary-copy\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-sys-fs\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-var-lib-kubelet\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340751    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40daca0-6d80-4669-b2de-895ea47b7087-host-var-lib-kubelet\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.340755    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/182a7a63-1fd0-452c-800f-e8866add61bf-sys-fs\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.341006    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a8440fa-890d-46a8-ae75-c0c25838ebe4-ovnkube-script-lib\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.341097    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a780d76b-930b-46f0-96c7-ffab4bc1166a-iptables-alerter-script\") pod \"iptables-alerter-6nmmk\" (UID: \"a780d76b-930b-46f0-96c7-ffab4bc1166a\") " pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.341145    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a40daca0-6d80-4669-b2de-895ea47b7087-cni-binary-copy\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.342531 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.342108    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a8440fa-890d-46a8-ae75-c0c25838ebe4-ovn-node-metrics-cert\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.358708 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.358657    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7qc\" (UniqueName: \"kubernetes.io/projected/66cf39f9-4496-49ad-aabc-e9453dab385a-kube-api-access-2b7qc\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:04.359792 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.359774    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:45:04.359863 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.359796    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:45:04.359863 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.359810    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m7fmb for pod openshift-network-diagnostics/network-check-target-zn2d9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:04.359863 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.359858    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb podName:d5ada156-013a-43b9-a444-d4fd7984a5be nodeName:}" failed. No retries permitted until 2026-07-02 13:45:04.859845912 +0000 UTC m=+4.701483443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fmb" (UniqueName: "kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb") pod "network-check-target-zn2d9" (UID: "d5ada156-013a-43b9-a444-d4fd7984a5be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:04.361028 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.361005    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvl2r\" (UniqueName: \"kubernetes.io/projected/182a7a63-1fd0-452c-800f-e8866add61bf-kube-api-access-xvl2r\") pod \"aws-ebs-csi-driver-node-ms4r7\" (UID: \"182a7a63-1fd0-452c-800f-e8866add61bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.361517 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.361494    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47c5\" (UniqueName: \"kubernetes.io/projected/a40daca0-6d80-4669-b2de-895ea47b7087-kube-api-access-j47c5\") pod \"multus-94mkh\" (UID: \"a40daca0-6d80-4669-b2de-895ea47b7087\") " pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.362331 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.362311    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2qc\" (UniqueName: \"kubernetes.io/projected/5a8440fa-890d-46a8-ae75-c0c25838ebe4-kube-api-access-4p2qc\") pod \"ovnkube-node-j86jl\" (UID: \"5a8440fa-890d-46a8-ae75-c0c25838ebe4\") " pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.362398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.362340    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sjnq\" (UniqueName: \"kubernetes.io/projected/a780d76b-930b-46f0-96c7-ffab4bc1166a-kube-api-access-5sjnq\") pod \"iptables-alerter-6nmmk\" (UID: \"a780d76b-930b-46f0-96c7-ffab4bc1166a\") " pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.426014 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.425989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rq799"
Jul 02 13:45:04.433492 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.433468    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a804050_c3a7_4cf3_923b_5146f3dde7c3.slice/crio-5827b34cdb74404c4319efa87d46109371531a84682c2df086856e6fba64fc2c WatchSource:0}: Error finding container 5827b34cdb74404c4319efa87d46109371531a84682c2df086856e6fba64fc2c: Status 404 returned error can't find the container with id 5827b34cdb74404c4319efa87d46109371531a84682c2df086856e6fba64fc2c
Jul 02 13:45:04.433492 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.433481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnjj7"
Jul 02 13:45:04.440694 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.440675    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095b3c78_4147_4098_b517_722b1e4941f1.slice/crio-d7c4ae70ea6abacac0dcdf23e14af0a4fd9741e1dccd8010a2574ee445ddfcdc WatchSource:0}: Error finding container d7c4ae70ea6abacac0dcdf23e14af0a4fd9741e1dccd8010a2574ee445ddfcdc: Status 404 returned error can't find the container with id d7c4ae70ea6abacac0dcdf23e14af0a4fd9741e1dccd8010a2574ee445ddfcdc
Jul 02 13:45:04.442795 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.442780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fpdcn"
Jul 02 13:45:04.447672 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.447653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:04.449537 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.449524    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31be0683_8d85_4b98_927f_7161f9326721.slice/crio-499cb7de6f66bf90656544b266711f530082a9c2bbc5f319b3d9ceeb5a76d167 WatchSource:0}: Error finding container 499cb7de6f66bf90656544b266711f530082a9c2bbc5f319b3d9ceeb5a76d167: Status 404 returned error can't find the container with id 499cb7de6f66bf90656544b266711f530082a9c2bbc5f319b3d9ceeb5a76d167
Jul 02 13:45:04.454714 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.454698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4hzt5"
Jul 02 13:45:04.454951 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.454933    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca65baf_3992_41ea_b4b9_e24681de33d3.slice/crio-19122f632b3a186b1d8cbd26fcc2eb099f29c7a3402c9cdeef694232d0ba0e69 WatchSource:0}: Error finding container 19122f632b3a186b1d8cbd26fcc2eb099f29c7a3402c9cdeef694232d0ba0e69: Status 404 returned error can't find the container with id 19122f632b3a186b1d8cbd26fcc2eb099f29c7a3402c9cdeef694232d0ba0e69
Jul 02 13:45:04.461996 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.461974    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-94mkh"
Jul 02 13:45:04.463304 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.463284    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a7dc11_0af2_4222_b6bf_06c30c9f50c0.slice/crio-340bb1b2b82b39aaf5a8ea8f4bf43eed618df0c0e760f87f54928741f7d5b546 WatchSource:0}: Error finding container 340bb1b2b82b39aaf5a8ea8f4bf43eed618df0c0e760f87f54928741f7d5b546: Status 404 returned error can't find the container with id 340bb1b2b82b39aaf5a8ea8f4bf43eed618df0c0e760f87f54928741f7d5b546
Jul 02 13:45:04.469417 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.469399    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6nmmk"
Jul 02 13:45:04.469982 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.469959    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40daca0_6d80_4669_b2de_895ea47b7087.slice/crio-bbcca795255d5eacfa334f2ce30e7be9f5a43e6302212a28d1b0b35fe5d647e9 WatchSource:0}: Error finding container bbcca795255d5eacfa334f2ce30e7be9f5a43e6302212a28d1b0b35fe5d647e9: Status 404 returned error can't find the container with id bbcca795255d5eacfa334f2ce30e7be9f5a43e6302212a28d1b0b35fe5d647e9
Jul 02 13:45:04.474984 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.474964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:04.478926 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.478009    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda780d76b_930b_46f0_96c7_ffab4bc1166a.slice/crio-d56d1e51ceddfb31aee60f00fd4affef43da1caed1de048ee29a74ec16d5d79c WatchSource:0}: Error finding container d56d1e51ceddfb31aee60f00fd4affef43da1caed1de048ee29a74ec16d5d79c: Status 404 returned error can't find the container with id d56d1e51ceddfb31aee60f00fd4affef43da1caed1de048ee29a74ec16d5d79c
Jul 02 13:45:04.479368 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.479163    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7"
Jul 02 13:45:04.487453 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.487427    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8440fa_890d_46a8_ae75_c0c25838ebe4.slice/crio-b901261c56cf8ea8d9e252fff6a7a1dfcf204d6be3cb184d7eeae70b48e89624 WatchSource:0}: Error finding container b901261c56cf8ea8d9e252fff6a7a1dfcf204d6be3cb184d7eeae70b48e89624: Status 404 returned error can't find the container with id b901261c56cf8ea8d9e252fff6a7a1dfcf204d6be3cb184d7eeae70b48e89624
Jul 02 13:45:04.488547 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:04.488408    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182a7a63_1fd0_452c_800f_e8866add61bf.slice/crio-93fe29d8c6ac24de4e1aac03370aedd6d1d796c8324f98753e9284fcfbbe73bf WatchSource:0}: Error finding container 93fe29d8c6ac24de4e1aac03370aedd6d1d796c8324f98753e9284fcfbbe73bf: Status 404 returned error can't find the container with id 93fe29d8c6ac24de4e1aac03370aedd6d1d796c8324f98753e9284fcfbbe73bf
Jul 02 13:45:04.502609 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.502564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wh4zs" event={"ID":"0ca65baf-3992-41ea-b4b9-e24681de33d3","Type":"ContainerStarted","Data":"19122f632b3a186b1d8cbd26fcc2eb099f29c7a3402c9cdeef694232d0ba0e69"}
Jul 02 13:45:04.503675 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.503645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnjj7" event={"ID":"095b3c78-4147-4098-b517-722b1e4941f1","Type":"ContainerStarted","Data":"d7c4ae70ea6abacac0dcdf23e14af0a4fd9741e1dccd8010a2574ee445ddfcdc"}
Jul 02 13:45:04.504985 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.504960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rq799" event={"ID":"2a804050-c3a7-4cf3-923b-5146f3dde7c3","Type":"ContainerStarted","Data":"5827b34cdb74404c4319efa87d46109371531a84682c2df086856e6fba64fc2c"}
Jul 02 13:45:04.506219 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.506197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal" event={"ID":"99227ef48a198d127b2226d0b66cb13b","Type":"ContainerStarted","Data":"fb305675cbb0354a71c250c77586344c350cde081c0c3ea2bc1b9d2679652f0c"}
Jul 02 13:45:04.507473 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.507448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal" event={"ID":"24dda35e4ab82f2be5688279542e5a7f","Type":"ContainerStarted","Data":"5bff47e895f82b8b6f16201676a1c38c7bf8081b72c6814068bc1090c2c12656"}
Jul 02 13:45:04.508658 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.508638    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7" event={"ID":"182a7a63-1fd0-452c-800f-e8866add61bf","Type":"ContainerStarted","Data":"93fe29d8c6ac24de4e1aac03370aedd6d1d796c8324f98753e9284fcfbbe73bf"}
Jul 02 13:45:04.509853 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.509833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-94mkh" event={"ID":"a40daca0-6d80-4669-b2de-895ea47b7087","Type":"ContainerStarted","Data":"bbcca795255d5eacfa334f2ce30e7be9f5a43e6302212a28d1b0b35fe5d647e9"}
Jul 02 13:45:04.511077 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.511054    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fpdcn" event={"ID":"31be0683-8d85-4b98-927f-7161f9326721","Type":"ContainerStarted","Data":"499cb7de6f66bf90656544b266711f530082a9c2bbc5f319b3d9ceeb5a76d167"}
Jul 02 13:45:04.512217 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.512197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"b901261c56cf8ea8d9e252fff6a7a1dfcf204d6be3cb184d7eeae70b48e89624"}
Jul 02 13:45:04.513453 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.513432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6nmmk" event={"ID":"a780d76b-930b-46f0-96c7-ffab4bc1166a","Type":"ContainerStarted","Data":"d56d1e51ceddfb31aee60f00fd4affef43da1caed1de048ee29a74ec16d5d79c"}
Jul 02 13:45:04.514674 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.514655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerStarted","Data":"340bb1b2b82b39aaf5a8ea8f4bf43eed618df0c0e760f87f54928741f7d5b546"}
Jul 02 13:45:04.557582 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.557562    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:45:04.845490 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.845419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:04.845659 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.845496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:04.845659 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.845588    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:04.845659 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.845645    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:04.845659 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.845652    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret podName:4ad7ad64-df1a-46f9-8e19-2b7333b838f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:05.845634549 +0000 UTC m=+5.687272046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret") pod "global-pull-secret-syncer-5mxmz" (UID: "4ad7ad64-df1a-46f9-8e19-2b7333b838f5") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:04.845887 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.845695    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs podName:66cf39f9-4496-49ad-aabc-e9453dab385a nodeName:}" failed. No retries permitted until 2026-07-02 13:45:05.845679309 +0000 UTC m=+5.687316805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs") pod "network-metrics-daemon-dm4g2" (UID: "66cf39f9-4496-49ad-aabc-e9453dab385a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:04.946310 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:04.946153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:04.946457 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.946359    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:45:04.946457 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.946379    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:45:04.946457 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.946403    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m7fmb for pod openshift-network-diagnostics/network-check-target-zn2d9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:04.946640 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:04.946469    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb podName:d5ada156-013a-43b9-a444-d4fd7984a5be nodeName:}" failed. No retries permitted until 2026-07-02 13:45:05.946449736 +0000 UTC m=+5.788087233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fmb" (UniqueName: "kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb") pod "network-check-target-zn2d9" (UID: "d5ada156-013a-43b9-a444-d4fd7984a5be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:05.181507 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:05.181389    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:40:04 +0000 UTC" deadline="2028-01-01 10:13:10.926226412 +0000 UTC"
Jul 02 13:45:05.181507 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:05.181429    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13148h28m5.7448017s"
Jul 02 13:45:05.498625 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:05.498551    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:05.499314 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:05.498892    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:05.499314 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.498979    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:05.499314 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.499031    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:05.856461 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:05.855452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:05.856461 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:05.855547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:05.856461 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.855828    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:05.856461 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.855906    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs podName:66cf39f9-4496-49ad-aabc-e9453dab385a nodeName:}" failed. No retries permitted until 2026-07-02 13:45:07.855884947 +0000 UTC m=+7.697522443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs") pod "network-metrics-daemon-dm4g2" (UID: "66cf39f9-4496-49ad-aabc-e9453dab385a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:05.856461 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.856332    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:05.856461 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.856383    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret podName:4ad7ad64-df1a-46f9-8e19-2b7333b838f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:07.856367467 +0000 UTC m=+7.698004963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret") pod "global-pull-secret-syncer-5mxmz" (UID: "4ad7ad64-df1a-46f9-8e19-2b7333b838f5") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:05.957094 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:05.956885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:05.957094 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.957081    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:45:05.957094 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.957101    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:45:05.957378 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.957115    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m7fmb for pod openshift-network-diagnostics/network-check-target-zn2d9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:05.957378 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:05.957172    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb podName:d5ada156-013a-43b9-a444-d4fd7984a5be nodeName:}" failed. No retries permitted until 2026-07-02 13:45:07.957154327 +0000 UTC m=+7.798791825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fmb" (UniqueName: "kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb") pod "network-check-target-zn2d9" (UID: "d5ada156-013a-43b9-a444-d4fd7984a5be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:06.182217 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:06.181997    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:40:04 +0000 UTC" deadline="2027-11-28 15:36:25.366855877 +0000 UTC"
Jul 02 13:45:06.182340 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:06.182222    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12337h51m19.184640799s"
Jul 02 13:45:06.499764 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:06.499644    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:06.500207 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:06.499816    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:06.566898 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:06.566115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal" event={"ID":"24dda35e4ab82f2be5688279542e5a7f","Type":"ContainerStarted","Data":"7523d8ae764779d84ac27271b91dd0dac5e7b44c2c7cd0d94f7661ef9c4f98e7"}
Jul 02 13:45:07.498184 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:07.498157    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:07.498359 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.498289    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:07.498428 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:07.498412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:07.498551 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.498514    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:07.872420 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:07.872352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:07.872803 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:07.872437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:07.872803 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.872523    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:07.872803 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.872586    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs podName:66cf39f9-4496-49ad-aabc-e9453dab385a nodeName:}" failed. No retries permitted until 2026-07-02 13:45:11.872567171 +0000 UTC m=+11.714204674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs") pod "network-metrics-daemon-dm4g2" (UID: "66cf39f9-4496-49ad-aabc-e9453dab385a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:07.872803 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.872613    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:07.872803 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.872685    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret podName:4ad7ad64-df1a-46f9-8e19-2b7333b838f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:11.872657045 +0000 UTC m=+11.714294539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret") pod "global-pull-secret-syncer-5mxmz" (UID: "4ad7ad64-df1a-46f9-8e19-2b7333b838f5") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:07.973766 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:07.973734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:07.973916 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.973897    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:45:07.973973 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.973918    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:45:07.973973 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.973930    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m7fmb for pod openshift-network-diagnostics/network-check-target-zn2d9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:07.974108 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:07.974003    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb podName:d5ada156-013a-43b9-a444-d4fd7984a5be nodeName:}" failed. No retries permitted until 2026-07-02 13:45:11.973975701 +0000 UTC m=+11.815613209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fmb" (UniqueName: "kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb") pod "network-check-target-zn2d9" (UID: "d5ada156-013a-43b9-a444-d4fd7984a5be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:08.498588 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:08.498557    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:08.498801 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:08.498691    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:08.574092 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:08.573979    2566 generic.go:356] "Generic (PLEG): container finished" podID="24dda35e4ab82f2be5688279542e5a7f" containerID="7523d8ae764779d84ac27271b91dd0dac5e7b44c2c7cd0d94f7661ef9c4f98e7" exitCode=0
Jul 02 13:45:08.574092 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:08.574056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal" event={"ID":"24dda35e4ab82f2be5688279542e5a7f","Type":"ContainerDied","Data":"7523d8ae764779d84ac27271b91dd0dac5e7b44c2c7cd0d94f7661ef9c4f98e7"}
Jul 02 13:45:09.498313 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:09.498283    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:09.498785 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:09.498417    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:09.498785 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:09.498500    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:09.498785 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:09.498601    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:10.498458 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:10.497973    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:10.498458 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:10.498116    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:11.498451 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:11.498421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:11.498705 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:11.498421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:11.498705 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:11.498553    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:11.498705 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:11.498611    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:11.906945 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:11.906215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:11.906945 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:11.906288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:11.906945 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:11.906403    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:11.906945 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:11.906462    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret podName:4ad7ad64-df1a-46f9-8e19-2b7333b838f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:19.906440873 +0000 UTC m=+19.748078366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret") pod "global-pull-secret-syncer-5mxmz" (UID: "4ad7ad64-df1a-46f9-8e19-2b7333b838f5") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:11.906945 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:11.906829    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:11.906945 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:11.906871    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs podName:66cf39f9-4496-49ad-aabc-e9453dab385a nodeName:}" failed. No retries permitted until 2026-07-02 13:45:19.906856639 +0000 UTC m=+19.748494134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs") pod "network-metrics-daemon-dm4g2" (UID: "66cf39f9-4496-49ad-aabc-e9453dab385a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:12.007425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:12.007386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:12.007618 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:12.007549    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:45:12.007618 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:12.007567    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:45:12.007618 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:12.007578    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m7fmb for pod openshift-network-diagnostics/network-check-target-zn2d9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:12.007782 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:12.007627    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb podName:d5ada156-013a-43b9-a444-d4fd7984a5be nodeName:}" failed. No retries permitted until 2026-07-02 13:45:20.007609839 +0000 UTC m=+19.849247331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fmb" (UniqueName: "kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb") pod "network-check-target-zn2d9" (UID: "d5ada156-013a-43b9-a444-d4fd7984a5be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:12.499406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:12.499332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:12.499877 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:12.499418    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:13.498298 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:13.498091    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:13.498298 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:13.498232    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:13.498589 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:13.498564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:13.498726 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:13.498670    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:14.497975 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:14.497944    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:14.498403 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:14.498084    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:15.498136 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:15.498113    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:15.498430 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:15.498113    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:15.498430 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:15.498208    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:15.498430 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:15.498278    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:16.498222 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:16.498190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:16.498621 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:16.498304    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:17.498111 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:17.498081    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:17.498279 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:17.498185    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:17.498279 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:17.498263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:17.498694 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:17.498378    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:18.498599 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:18.498571    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:18.499003 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:18.498681    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:19.498524 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:19.498496    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:19.498743 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:19.498506    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:19.498743 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:19.498609    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:19.498743 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:19.498698    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:19.977959 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:19.977898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:19.977959 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:19.977949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:19.978176 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:19.978072    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:19.978176 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:19.978125    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs podName:66cf39f9-4496-49ad-aabc-e9453dab385a nodeName:}" failed. No retries permitted until 2026-07-02 13:45:35.978111364 +0000 UTC m=+35.819748855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs") pod "network-metrics-daemon-dm4g2" (UID: "66cf39f9-4496-49ad-aabc-e9453dab385a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:19.978176 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:19.978072    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:19.978295 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:19.978202    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret podName:4ad7ad64-df1a-46f9-8e19-2b7333b838f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:35.978189842 +0000 UTC m=+35.819827344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret") pod "global-pull-secret-syncer-5mxmz" (UID: "4ad7ad64-df1a-46f9-8e19-2b7333b838f5") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:20.078731 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:20.078702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:20.078877 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:20.078804    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:45:20.078877 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:20.078816    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:45:20.078877 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:20.078824    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m7fmb for pod openshift-network-diagnostics/network-check-target-zn2d9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:20.078877 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:20.078861    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb podName:d5ada156-013a-43b9-a444-d4fd7984a5be nodeName:}" failed. No retries permitted until 2026-07-02 13:45:36.078849239 +0000 UTC m=+35.920486731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fmb" (UniqueName: "kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb") pod "network-check-target-zn2d9" (UID: "d5ada156-013a-43b9-a444-d4fd7984a5be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:20.498052 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:20.498001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:20.498215 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:20.498128    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:21.498127 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:21.498098    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:21.498566 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:21.498205    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:21.498566 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:21.498274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:21.498566 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:21.498393    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:22.498832 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:22.498800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:22.499260 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:22.498914    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:23.498493 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:23.498445    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:23.498682 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:23.498445    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:23.498682 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:23.498598    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:23.498804 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:23.498715    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:24.498201 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.498177    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:24.498553 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:24.498293    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:24.603828 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.603803    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"2dde2b2c41c2edee6ff4656f4512fc87773ff34a2cace1b07df14deb0dc62173"}
Jul 02 13:45:24.603943 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.603863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"6e494dad7b5bb70120ac34fc2ac9eb93ec870133cd80e77b39a4e0eefcc9f1c2"}
Jul 02 13:45:24.605271 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.605250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerStarted","Data":"b0bd38ae00fc29331e86f1dc561f381e8b576671b08b337c317d7df7099744b0"}
Jul 02 13:45:24.606583 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.606564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wh4zs" event={"ID":"0ca65baf-3992-41ea-b4b9-e24681de33d3","Type":"ContainerStarted","Data":"35ea59f245cf53bd658f0e9db45436f1b93a38d84c3e81963e0067cde16fc80f"}
Jul 02 13:45:24.607943 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.607923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnjj7" event={"ID":"095b3c78-4147-4098-b517-722b1e4941f1","Type":"ContainerStarted","Data":"bb872a52dcd3639b747d99a5266179f7e60c3c204d79cc494b69d131d5e23754"}
Jul 02 13:45:24.609306 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.609281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rq799" event={"ID":"2a804050-c3a7-4cf3-923b-5146f3dde7c3","Type":"ContainerStarted","Data":"b68742e5329ba94c6a91ad78711957cc139252aa84df3e33092bacc5d27c199e"}
Jul 02 13:45:24.610888 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.610865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal" event={"ID":"99227ef48a198d127b2226d0b66cb13b","Type":"ContainerStarted","Data":"40fc29afc2bd636c44970f47cdca741e15ac677dd0d0874e1920a21a8b98a7b3"}
Jul 02 13:45:24.612537 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.612509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal" event={"ID":"24dda35e4ab82f2be5688279542e5a7f","Type":"ContainerStarted","Data":"733d6c7625b173af3cc88efc593a6b01aef148b3deb2a3d5ea450a9b1612060a"}
Jul 02 13:45:24.613794 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.613773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7" event={"ID":"182a7a63-1fd0-452c-800f-e8866add61bf","Type":"ContainerStarted","Data":"c4119f37355fb9be35a1693bb2d307c1bc03571ebad7b814b7ad3e9ae4268207"}
Jul 02 13:45:24.615077 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.615055    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fpdcn" event={"ID":"31be0683-8d85-4b98-927f-7161f9326721","Type":"ContainerStarted","Data":"98ecea3e9b58949451acfa1d439d75b2f2b2bc0d3ccfba632f9e585a578232df"}
Jul 02 13:45:24.642728 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.642690    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rq799" podStartSLOduration=2.962832187 podStartE2EDuration="22.642678408s" totalImagesPullingTime="19.679846221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:24.642678263 +0000 UTC m=+24.484315779" watchObservedRunningTime="2026-07-02 13:45:24.642678408 +0000 UTC m=+24.484315921"
Jul 02 13:45:24.659005 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.658970    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-2.ec2.internal" podStartSLOduration=20.658958596 podStartE2EDuration="20.658958596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:24.658703909 +0000 UTC m=+24.500341423" watchObservedRunningTime="2026-07-02 13:45:24.658958596 +0000 UTC m=+24.500596110"
Jul 02 13:45:24.671947 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.671911    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-2.ec2.internal" podStartSLOduration=20.671895283 podStartE2EDuration="20.671895283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:24.671695332 +0000 UTC m=+24.513332845" watchObservedRunningTime="2026-07-02 13:45:24.671895283 +0000 UTC m=+24.513532797"
Jul 02 13:45:24.687125 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.687089    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tnjj7" podStartSLOduration=11.55895734 podStartE2EDuration="22.687075442s" totalImagesPullingTime="11.128118102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:24.686101381 +0000 UTC m=+24.527738896" watchObservedRunningTime="2026-07-02 13:45:24.687075442 +0000 UTC m=+24.528712957"
Jul 02 13:45:24.703710 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.703653    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fpdcn" podStartSLOduration=2.956321198 podStartE2EDuration="22.703642893s" totalImagesPullingTime="19.747321695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:24.703442615 +0000 UTC m=+24.545080128" watchObservedRunningTime="2026-07-02 13:45:24.703642893 +0000 UTC m=+24.545280405"
Jul 02 13:45:24.724326 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:24.724297    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wh4zs" podStartSLOduration=3.4675684 podStartE2EDuration="22.724287847s" totalImagesPullingTime="19.256719447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:24.723876661 +0000 UTC m=+24.565514174" watchObservedRunningTime="2026-07-02 13:45:24.724287847 +0000 UTC m=+24.565925360"
Jul 02 13:45:25.498153 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.497965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:25.498339 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.498033    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:25.498339 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:25.498228    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:25.498339 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:25.498309    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:25.618739 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.618715    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-94mkh" event={"ID":"a40daca0-6d80-4669-b2de-895ea47b7087","Type":"ContainerStarted","Data":"456ac4a9b75c971357c5e5e9a30d8df45b1820c4136218e3d3f8e7e84ee00d4e"}
Jul 02 13:45:25.621891 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.621872    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 13:45:25.622280 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.622260    2566 generic.go:356] "Generic (PLEG): container finished" podID="5a8440fa-890d-46a8-ae75-c0c25838ebe4" containerID="2dde2b2c41c2edee6ff4656f4512fc87773ff34a2cace1b07df14deb0dc62173" exitCode=1
Jul 02 13:45:25.622359 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.622336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerDied","Data":"2dde2b2c41c2edee6ff4656f4512fc87773ff34a2cace1b07df14deb0dc62173"}
Jul 02 13:45:25.622433 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.622416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"426a2b6760ab6c7f4c5898a9c1d31fb93b7bd0a8113d8d38083e03823150ee29"}
Jul 02 13:45:25.622510 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.622475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"82906298711cb4d1ded9b1b20bdd7cd5956bb25158f9e55d31d70e57c8ca8b3e"}
Jul 02 13:45:25.622571 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.622537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"28b552cb5fa68bdca95b065dbc1333052843028e478bdb10ccc8287a50b8f769"}
Jul 02 13:45:25.622621 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.622576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"ececf07f10e9c70a4466b9c7b12d276798c119bdbc831011ad239f27f7bf3933"}
Jul 02 13:45:25.623943 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.623920    2566 generic.go:356] "Generic (PLEG): container finished" podID="04a7dc11-0af2-4222-b6bf-06c30c9f50c0" containerID="b0bd38ae00fc29331e86f1dc561f381e8b576671b08b337c317d7df7099744b0" exitCode=0
Jul 02 13:45:25.624032 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.624005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerDied","Data":"b0bd38ae00fc29331e86f1dc561f381e8b576671b08b337c317d7df7099744b0"}
Jul 02 13:45:25.637028 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.636989    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-94mkh" podStartSLOduration=2.351290079 podStartE2EDuration="22.636978091s" totalImagesPullingTime="20.285688012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:25.63639352 +0000 UTC m=+25.478031156" watchObservedRunningTime="2026-07-02 13:45:25.636978091 +0000 UTC m=+25.478615603"
Jul 02 13:45:25.690732 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:25.690710    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 13:45:26.498736 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:26.498708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:26.499271 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:26.498829    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:26.509269 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:26.509189    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T13:45:25.690729057Z","UUID":"f3afdf41-3803-4107-a685-c0ab6d1ae223","Handler":null,"Name":"","Endpoint":""}
Jul 02 13:45:26.510936 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:26.510910    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 13:45:26.511062 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:26.510954    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 13:45:26.628443 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:26.628414    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6nmmk" event={"ID":"a780d76b-930b-46f0-96c7-ffab4bc1166a","Type":"ContainerStarted","Data":"5a72ac08a2ef955a3f9d675db2e4017b077f302c453e11e8b7e021e4e88b7ed7"}
Jul 02 13:45:26.630568 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:26.630544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7" event={"ID":"182a7a63-1fd0-452c-800f-e8866add61bf","Type":"ContainerStarted","Data":"c9bf50428455b9265e683059b32e75485a349c18112ef963c5e76c1671437ce7"}
Jul 02 13:45:26.643761 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:26.643723    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6nmmk" podStartSLOduration=3.911229619 podStartE2EDuration="23.643706414s" totalImagesPullingTime="19.732476795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:26.643241628 +0000 UTC m=+26.484879144" watchObservedRunningTime="2026-07-02 13:45:26.643706414 +0000 UTC m=+26.485343928"
Jul 02 13:45:27.498884 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:27.498664    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:27.498884 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:27.498739    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:27.499400 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:27.498942    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:27.499400 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:27.499083    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:27.635575 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:27.635493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7" event={"ID":"182a7a63-1fd0-452c-800f-e8866add61bf","Type":"ContainerStarted","Data":"0e1cb4c5d1ae7aa0a5d5f6b05d5626f5740d5ce4e769f62c09b4473683ec39d3"}
Jul 02 13:45:27.638871 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:27.638848    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 13:45:27.639502 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:27.639458    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"9f04d1db1075662750f8bae2212b4ae71e278805f0931648c16b991617f442fd"}
Jul 02 13:45:28.498188 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:28.498156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:28.498398 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:28.498285    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:29.255699 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.255667    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:29.256310 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.256230    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:29.273542 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.273497    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms4r7" podStartSLOduration=4.361699606 podStartE2EDuration="26.273463657s" totalImagesPullingTime="21.911764051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:27.657334371 +0000 UTC m=+27.498971884" watchObservedRunningTime="2026-07-02 13:45:29.273463657 +0000 UTC m=+29.115101172"
Jul 02 13:45:29.497952 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.497903    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:29.498201 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:29.498007    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:29.498201 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.498093    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:29.498387 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:29.498205    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:29.647461 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.647301    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 13:45:29.647888 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.647851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"f95eed6923582f9b3f4a61f5750bc199802c6d4fec68c3b0771d9de459a0fa8d"}
Jul 02 13:45:29.648234 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.648215    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:29.648373 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.648360    2566 scope.go:117] "RemoveContainer" containerID="2dde2b2c41c2edee6ff4656f4512fc87773ff34a2cace1b07df14deb0dc62173"
Jul 02 13:45:29.649948 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.649925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerStarted","Data":"220d968498a4c901b610609c1cbdf2b452d0325d2e079ffa5888020031b5bdcb"}
Jul 02 13:45:29.650172 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.650147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:29.650752 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.650714    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wh4zs"
Jul 02 13:45:29.663230 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:29.663203    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:30.498142 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:30.498114    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:30.498685 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:30.498203    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:30.656530 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:30.656442    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 13:45:30.656903 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:30.656871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" event={"ID":"5a8440fa-890d-46a8-ae75-c0c25838ebe4","Type":"ContainerStarted","Data":"4fc8937322ce2ea2689a01715ffd760388cf43fa3a9e8b2e670296207a48451f"}
Jul 02 13:45:30.657226 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:30.657200    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:30.657390 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:30.657371    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:30.659857 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:30.659829    2566 generic.go:356] "Generic (PLEG): container finished" podID="04a7dc11-0af2-4222-b6bf-06c30c9f50c0" containerID="220d968498a4c901b610609c1cbdf2b452d0325d2e079ffa5888020031b5bdcb" exitCode=0
Jul 02 13:45:30.660840 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:30.660809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerDied","Data":"220d968498a4c901b610609c1cbdf2b452d0325d2e079ffa5888020031b5bdcb"}
Jul 02 13:45:30.679215 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:30.679191    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:45:30.693139 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:30.693096    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl" podStartSLOduration=7.97980313 podStartE2EDuration="27.693081748s" totalImagesPullingTime="19.713278618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:30.692445936 +0000 UTC m=+30.534083449" watchObservedRunningTime="2026-07-02 13:45:30.693081748 +0000 UTC m=+30.534719281"
Jul 02 13:45:31.497939 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:31.497914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:31.498108 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:31.497954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:31.498108 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:31.498058    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:31.498389 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:31.498177    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:31.577094 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:31.576924    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5mxmz"]
Jul 02 13:45:31.577202 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:31.577187    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:31.577320 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:31.577298    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:31.580711 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:31.580687    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dm4g2"]
Jul 02 13:45:31.583520 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:31.583497    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zn2d9"]
Jul 02 13:45:31.661392 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:31.661372    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:31.661506 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:31.661379    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:31.661576 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:31.661527    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:31.661633 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:31.661614    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:32.665457 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:32.665408    2566 generic.go:356] "Generic (PLEG): container finished" podID="04a7dc11-0af2-4222-b6bf-06c30c9f50c0" containerID="c35408b7e48130bed05d3a5edf5780140760e23b912f6864886b354ce510641b" exitCode=0
Jul 02 13:45:32.665865 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:32.665511    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerDied","Data":"c35408b7e48130bed05d3a5edf5780140760e23b912f6864886b354ce510641b"}
Jul 02 13:45:33.497975 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:33.497906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:33.497975 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:33.497939    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:33.497975 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:33.497952    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:33.498185 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:33.498075    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:33.498221 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:33.498182    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:33.498266 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:33.498250    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:34.672268 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:34.672239    2566 generic.go:356] "Generic (PLEG): container finished" podID="04a7dc11-0af2-4222-b6bf-06c30c9f50c0" containerID="2490689b7c7a27f16f1c3a31f28fde55a113c3e1de1c0d25cda29e1a5acdbeb5" exitCode=0
Jul 02 13:45:34.672986 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:34.672294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerDied","Data":"2490689b7c7a27f16f1c3a31f28fde55a113c3e1de1c0d25cda29e1a5acdbeb5"}
Jul 02 13:45:35.498438 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:35.498363    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:35.498438 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:35.498388    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:35.498680 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:35.498509    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn2d9" podUID="d5ada156-013a-43b9-a444-d4fd7984a5be"
Jul 02 13:45:35.498680 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:35.498579    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm4g2" podUID="66cf39f9-4496-49ad-aabc-e9453dab385a"
Jul 02 13:45:35.498680 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:35.498630    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:35.498818 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:35.498716    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mxmz" podUID="4ad7ad64-df1a-46f9-8e19-2b7333b838f5"
Jul 02 13:45:36.003735 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:36.003708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:36.004192 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:36.003757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:36.004192 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:36.003863    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:36.004192 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:36.003906    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:36.004192 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:36.003919    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret podName:4ad7ad64-df1a-46f9-8e19-2b7333b838f5 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:08.003900994 +0000 UTC m=+67.845538486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret") pod "global-pull-secret-syncer-5mxmz" (UID: "4ad7ad64-df1a-46f9-8e19-2b7333b838f5") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:45:36.004192 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:36.004183    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs podName:66cf39f9-4496-49ad-aabc-e9453dab385a nodeName:}" failed. No retries permitted until 2026-07-02 13:46:08.003964178 +0000 UTC m=+67.845601682 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs") pod "network-metrics-daemon-dm4g2" (UID: "66cf39f9-4496-49ad-aabc-e9453dab385a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:45:36.104596 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:36.104568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:36.104752 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:36.104735    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:45:36.104804 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:36.104763    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:45:36.104804 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:36.104776    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m7fmb for pod openshift-network-diagnostics/network-check-target-zn2d9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:36.104880 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:36.104842    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb podName:d5ada156-013a-43b9-a444-d4fd7984a5be nodeName:}" failed. No retries permitted until 2026-07-02 13:46:08.10481322 +0000 UTC m=+67.946450718 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fmb" (UniqueName: "kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb") pod "network-check-target-zn2d9" (UID: "d5ada156-013a-43b9-a444-d4fd7984a5be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:45:37.446659 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.446593    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-2.ec2.internal" event="NodeReady"
Jul 02 13:45:37.447015 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.446718    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 13:45:37.497877 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.497850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:45:37.498055 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.497850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:45:37.498116 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.497858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:45:37.505946 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.505871    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:45:37.506621 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.506598    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:45:37.506719 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.506697    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:45:37.506790 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.506725    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:45:37.506790 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.506739    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9jh8m\""
Jul 02 13:45:37.507002 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.506987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nskqf\""
Jul 02 13:45:37.652915 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.652885    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qkjj8"]
Jul 02 13:45:37.655120 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.655097    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2ljkk"]
Jul 02 13:45:37.655230 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.655200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.656969 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.656949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2ljkk"
Jul 02 13:45:37.667887 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.667868    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 13:45:37.667978 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.667949    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t9b9z\""
Jul 02 13:45:37.669133 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.669117    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 13:45:37.669632 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.669614    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 13:45:37.670669 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.670652    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 13:45:37.670763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.670675    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jj27z\""
Jul 02 13:45:37.670814 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.670785    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 13:45:37.701679 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.701630    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2ljkk"]
Jul 02 13:45:37.716419 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.716370    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qkjj8"]
Jul 02 13:45:37.718037 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.717996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aedca59b-0e89-4e5b-9092-3628bfc55297-tmp-dir\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.718136 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.718067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aedca59b-0e89-4e5b-9092-3628bfc55297-config-volume\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.718136 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.718092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eaabca2-d6be-46b6-9b5d-d7db2490f413-cert\") pod \"ingress-canary-2ljkk\" (UID: \"8eaabca2-d6be-46b6-9b5d-d7db2490f413\") " pod="openshift-ingress-canary/ingress-canary-2ljkk"
Jul 02 13:45:37.718136 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.718118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgl5\" (UniqueName: \"kubernetes.io/projected/8eaabca2-d6be-46b6-9b5d-d7db2490f413-kube-api-access-5bgl5\") pod \"ingress-canary-2ljkk\" (UID: \"8eaabca2-d6be-46b6-9b5d-d7db2490f413\") " pod="openshift-ingress-canary/ingress-canary-2ljkk"
Jul 02 13:45:37.718287 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.718182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.718287 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.718228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j889\" (UniqueName: \"kubernetes.io/projected/aedca59b-0e89-4e5b-9092-3628bfc55297-kube-api-access-5j889\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.818977 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.818947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j889\" (UniqueName: \"kubernetes.io/projected/aedca59b-0e89-4e5b-9092-3628bfc55297-kube-api-access-5j889\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.819088 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.819015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aedca59b-0e89-4e5b-9092-3628bfc55297-tmp-dir\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.819088 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.819049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aedca59b-0e89-4e5b-9092-3628bfc55297-config-volume\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.819088 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.819069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eaabca2-d6be-46b6-9b5d-d7db2490f413-cert\") pod \"ingress-canary-2ljkk\" (UID: \"8eaabca2-d6be-46b6-9b5d-d7db2490f413\") " pod="openshift-ingress-canary/ingress-canary-2ljkk"
Jul 02 13:45:37.819251 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.819096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgl5\" (UniqueName: \"kubernetes.io/projected/8eaabca2-d6be-46b6-9b5d-d7db2490f413-kube-api-access-5bgl5\") pod \"ingress-canary-2ljkk\" (UID: \"8eaabca2-d6be-46b6-9b5d-d7db2490f413\") " pod="openshift-ingress-canary/ingress-canary-2ljkk"
Jul 02 13:45:37.819251 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.819134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.819251 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:37.819239    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:45:37.819415 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:37.819288    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls podName:aedca59b-0e89-4e5b-9092-3628bfc55297 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:38.319271946 +0000 UTC m=+38.160909450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls") pod "dns-default-qkjj8" (UID: "aedca59b-0e89-4e5b-9092-3628bfc55297") : secret "dns-default-metrics-tls" not found
Jul 02 13:45:37.819415 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.819371    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aedca59b-0e89-4e5b-9092-3628bfc55297-tmp-dir\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.819840 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.819800    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aedca59b-0e89-4e5b-9092-3628bfc55297-config-volume\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.823766 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.823594    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8eaabca2-d6be-46b6-9b5d-d7db2490f413-cert\") pod \"ingress-canary-2ljkk\" (UID: \"8eaabca2-d6be-46b6-9b5d-d7db2490f413\") " pod="openshift-ingress-canary/ingress-canary-2ljkk"
Jul 02 13:45:37.836456 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.836430    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j889\" (UniqueName: \"kubernetes.io/projected/aedca59b-0e89-4e5b-9092-3628bfc55297-kube-api-access-5j889\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:37.848767 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.848745    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgl5\" (UniqueName: \"kubernetes.io/projected/8eaabca2-d6be-46b6-9b5d-d7db2490f413-kube-api-access-5bgl5\") pod \"ingress-canary-2ljkk\" (UID: \"8eaabca2-d6be-46b6-9b5d-d7db2490f413\") " pod="openshift-ingress-canary/ingress-canary-2ljkk"
Jul 02 13:45:37.971738 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:37.971656    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2ljkk"
Jul 02 13:45:38.322163 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:38.322085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:38.322318 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:38.322196    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:45:38.322318 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:38.322259    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls podName:aedca59b-0e89-4e5b-9092-3628bfc55297 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:39.322239414 +0000 UTC m=+39.163876907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls") pod "dns-default-qkjj8" (UID: "aedca59b-0e89-4e5b-9092-3628bfc55297") : secret "dns-default-metrics-tls" not found
Jul 02 13:45:39.327759 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:39.327722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:39.328053 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:39.327868    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:45:39.328053 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:39.327941    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls podName:aedca59b-0e89-4e5b-9092-3628bfc55297 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:41.327925468 +0000 UTC m=+41.169562959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls") pod "dns-default-qkjj8" (UID: "aedca59b-0e89-4e5b-9092-3628bfc55297") : secret "dns-default-metrics-tls" not found
Jul 02 13:45:39.455176 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:39.455153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2ljkk"]
Jul 02 13:45:39.459253 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:45:39.459229    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eaabca2_d6be_46b6_9b5d_d7db2490f413.slice/crio-a631cdd467f7a9f0c2899848c258347b9cd1315c516edc44a52263e9f9b72201 WatchSource:0}: Error finding container a631cdd467f7a9f0c2899848c258347b9cd1315c516edc44a52263e9f9b72201: Status 404 returned error can't find the container with id a631cdd467f7a9f0c2899848c258347b9cd1315c516edc44a52263e9f9b72201
Jul 02 13:45:39.685577 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:39.685518    2566 generic.go:356] "Generic (PLEG): container finished" podID="04a7dc11-0af2-4222-b6bf-06c30c9f50c0" containerID="ed444b1eacd0a9ecfda341dc2e0e9842b5c965a6c399d11db070116ce81e3332" exitCode=0
Jul 02 13:45:39.685721 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:39.685600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerDied","Data":"ed444b1eacd0a9ecfda341dc2e0e9842b5c965a6c399d11db070116ce81e3332"}
Jul 02 13:45:39.686665 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:39.686641    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2ljkk" event={"ID":"8eaabca2-d6be-46b6-9b5d-d7db2490f413","Type":"ContainerStarted","Data":"a631cdd467f7a9f0c2899848c258347b9cd1315c516edc44a52263e9f9b72201"}
Jul 02 13:45:40.692941 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:40.692718    2566 generic.go:356] "Generic (PLEG): container finished" podID="04a7dc11-0af2-4222-b6bf-06c30c9f50c0" containerID="10583dc3ed7b13d6d41bf65b26bcedfe149c2f78e97b7ec70b4077d88b8340f2" exitCode=0
Jul 02 13:45:40.692941 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:40.692804    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerDied","Data":"10583dc3ed7b13d6d41bf65b26bcedfe149c2f78e97b7ec70b4077d88b8340f2"}
Jul 02 13:45:41.339467 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:41.339401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:41.339602 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:41.339563    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:45:41.339649 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:41.339637    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls podName:aedca59b-0e89-4e5b-9092-3628bfc55297 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:45.339618169 +0000 UTC m=+45.181255664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls") pod "dns-default-qkjj8" (UID: "aedca59b-0e89-4e5b-9092-3628bfc55297") : secret "dns-default-metrics-tls" not found
Jul 02 13:45:41.699352 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:41.699317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" event={"ID":"04a7dc11-0af2-4222-b6bf-06c30c9f50c0","Type":"ContainerStarted","Data":"afb17f11f5f92f9865af385e0869edf63968fe9dc888b222c5b7d1241b83ac2b"}
Jul 02 13:45:41.726804 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:41.726752    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4hzt5" podStartSLOduration=9.491431492 podStartE2EDuration="39.726733288s" totalImagesPullingTime="30.235301796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:41.725183269 +0000 UTC m=+41.566820782" watchObservedRunningTime="2026-07-02 13:45:41.726733288 +0000 UTC m=+41.568370804"
Jul 02 13:45:43.704424 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:43.704386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2ljkk" event={"ID":"8eaabca2-d6be-46b6-9b5d-d7db2490f413","Type":"ContainerStarted","Data":"39cfc6417d09889e8cfc4b584203fe43d9db66af6b729fe9c34eb9ebe2651bc2"}
Jul 02 13:45:43.722931 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:43.722863    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2ljkk" podStartSLOduration=3.378923554 podStartE2EDuration="6.722850057s" totalImagesPullingTime="3.343926503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:43.722138991 +0000 UTC m=+43.563776509" watchObservedRunningTime="2026-07-02 13:45:43.722850057 +0000 UTC m=+43.564487570"
Jul 02 13:45:45.366959 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:45.366917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:45.367339 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:45.367034    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:45:45.367339 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:45.367085    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls podName:aedca59b-0e89-4e5b-9092-3628bfc55297 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:53.367071915 +0000 UTC m=+53.208709412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls") pod "dns-default-qkjj8" (UID: "aedca59b-0e89-4e5b-9092-3628bfc55297") : secret "dns-default-metrics-tls" not found
Jul 02 13:45:53.420619 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:45:53.420583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:45:53.421067 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:53.420731    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:45:53.421067 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:45:53.420789    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls podName:aedca59b-0e89-4e5b-9092-3628bfc55297 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:09.420774766 +0000 UTC m=+69.262412259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls") pod "dns-default-qkjj8" (UID: "aedca59b-0e89-4e5b-9092-3628bfc55297") : secret "dns-default-metrics-tls" not found
Jul 02 13:46:02.679971 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:02.679943    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j86jl"
Jul 02 13:46:05.259261 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.259233    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9"]
Jul 02 13:46:05.262431 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.262416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9"
Jul 02 13:46:05.264950 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.264930    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mvrbs\""
Jul 02 13:46:05.265627 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.265607    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 02 13:46:05.265713 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.265627    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:46:05.272087 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.272067    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9"]
Jul 02 13:46:05.297261 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.297229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xp6g\" (UniqueName: \"kubernetes.io/projected/932531fc-a0e6-4e3a-a5ea-46a1357a6409-kube-api-access-6xp6g\") pod \"volume-data-source-validator-69d9f679db-cl6t9\" (UID: \"932531fc-a0e6-4e3a-a5ea-46a1357a6409\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9"
Jul 02 13:46:05.370738 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.370712    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84fb55f8df-spq4p"]
Jul 02 13:46:05.372587 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.372573    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"]
Jul 02 13:46:05.372715 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.372699    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.374508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.374477    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:05.383379 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.383358    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 13:46:05.383516 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.383385    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 13:46:05.383516 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.383390    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 13:46:05.383516 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.383428    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 02 13:46:05.383516 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.383466    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 13:46:05.383847 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.383826    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bjcws\""
Jul 02 13:46:05.384383 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.384354    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 02 13:46:05.387448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.387432    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p922t\""
Jul 02 13:46:05.387562 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.387548    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 13:46:05.397821 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.397803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xp6g\" (UniqueName: \"kubernetes.io/projected/932531fc-a0e6-4e3a-a5ea-46a1357a6409-kube-api-access-6xp6g\") pod \"volume-data-source-validator-69d9f679db-cl6t9\" (UID: \"932531fc-a0e6-4e3a-a5ea-46a1357a6409\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9"
Jul 02 13:46:05.400169 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.400152    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84fb55f8df-spq4p"]
Jul 02 13:46:05.401403 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.401387    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 13:46:05.411940 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.411923    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"]
Jul 02 13:46:05.437277 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.437246    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xp6g\" (UniqueName: \"kubernetes.io/projected/932531fc-a0e6-4e3a-a5ea-46a1357a6409-kube-api-access-6xp6g\") pod \"volume-data-source-validator-69d9f679db-cl6t9\" (UID: \"932531fc-a0e6-4e3a-a5ea-46a1357a6409\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9"
Jul 02 13:46:05.466948 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.466920    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"]
Jul 02 13:46:05.468897 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.468879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.473532 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.473507    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 02 13:46:05.473625 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.473564    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wdb88\""
Jul 02 13:46:05.473625 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.473576    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 02 13:46:05.474050 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.474035    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 02 13:46:05.474429 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.474407    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:46:05.484883 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.484866    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"]
Jul 02 13:46:05.486610 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.486596    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:05.488438 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.488408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"]
Jul 02 13:46:05.489321 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.489306    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 02 13:46:05.489435 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.489418    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:46:05.490664 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.490647    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 02 13:46:05.491293 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.491270    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xjd7q\""
Jul 02 13:46:05.498237 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-certificates\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.498330 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-installation-pull-secrets\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.498330 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-bound-sa-token\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.498412 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.498412 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9a4f6553-c51e-49aa-b0b5-dee8541cee26-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:05.498510 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-image-registry-private-configuration\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.498510 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498428    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef9709a9-9047-412d-a162-071bb4647e73-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-6wr2h\" (UID: \"ef9709a9-9047-412d-a162-071bb4647e73\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.498510 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-trusted-ca\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.498510 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bb2x\" (UniqueName: \"kubernetes.io/projected/9a4f6553-c51e-49aa-b0b5-dee8541cee26-kube-api-access-6bb2x\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:05.498684 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:05.498684 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-ca-trust-extracted\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.498684 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rsj7\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-kube-api-access-6rsj7\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.498684 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef9709a9-9047-412d-a162-071bb4647e73-config\") pod \"service-ca-operator-5d7b7dc95c-6wr2h\" (UID: \"ef9709a9-9047-412d-a162-071bb4647e73\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.498684 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.498617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grps\" (UniqueName: \"kubernetes.io/projected/ef9709a9-9047-412d-a162-071bb4647e73-kube-api-access-7grps\") pod \"service-ca-operator-5d7b7dc95c-6wr2h\" (UID: \"ef9709a9-9047-412d-a162-071bb4647e73\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.505238 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.505215    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"]
Jul 02 13:46:05.546846 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.546781    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"]
Jul 02 13:46:05.569454 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.569430    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"]
Jul 02 13:46:05.569607 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.569558    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.570656 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.570635    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9"
Jul 02 13:46:05.572137 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.572111    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:46:05.572237 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.572156    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p2sl4\""
Jul 02 13:46:05.572237 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.572209    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 02 13:46:05.572435 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.572416    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 02 13:46:05.572541 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.572453    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 02 13:46:05.599354 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bb2x\" (UniqueName: \"kubernetes.io/projected/9a4f6553-c51e-49aa-b0b5-dee8541cee26-kube-api-access-6bb2x\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:05.599500 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjrm\" (UniqueName: \"kubernetes.io/projected/e3d9299e-86c3-4cf4-9464-6a3cb72e0287-kube-api-access-mzjrm\") pod \"kube-storage-version-migrator-operator-79964ff6f9-v7nvl\" (UID: \"e3d9299e-86c3-4cf4-9464-6a3cb72e0287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.599500 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599396    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pvhg\" (UniqueName: \"kubernetes.io/projected/32e69962-a074-45d3-8ab6-85bf35ad9801-kube-api-access-7pvhg\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:05.599500 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:05.599670 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-ca-trust-extracted\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.599670 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:05.599568    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:05.599670 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3d9299e-86c3-4cf4-9464-6a3cb72e0287-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-v7nvl\" (UID: \"e3d9299e-86c3-4cf4-9464-6a3cb72e0287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.599670 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsj7\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-kube-api-access-6rsj7\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.599670 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:05.599630    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls podName:9a4f6553-c51e-49aa-b0b5-dee8541cee26 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:06.09961008 +0000 UTC m=+65.941247575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-w5g2b" (UID: "9a4f6553-c51e-49aa-b0b5-dee8541cee26") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:05.599670 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef9709a9-9047-412d-a162-071bb4647e73-config\") pod \"service-ca-operator-5d7b7dc95c-6wr2h\" (UID: \"ef9709a9-9047-412d-a162-071bb4647e73\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.599966 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grps\" (UniqueName: \"kubernetes.io/projected/ef9709a9-9047-412d-a162-071bb4647e73-kube-api-access-7grps\") pod \"service-ca-operator-5d7b7dc95c-6wr2h\" (UID: \"ef9709a9-9047-412d-a162-071bb4647e73\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.599966 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:05.599966 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-certificates\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.599966 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-installation-pull-secrets\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.599966 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-bound-sa-token\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.599966 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599863    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-ca-trust-extracted\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.599966 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.599966 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9a4f6553-c51e-49aa-b0b5-dee8541cee26-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:05.599966 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-image-registry-private-configuration\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.600380 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:05.599977    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:46:05.600380 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.599986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef9709a9-9047-412d-a162-071bb4647e73-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-6wr2h\" (UID: \"ef9709a9-9047-412d-a162-071bb4647e73\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.600380 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:05.599997    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fb55f8df-spq4p: secret "image-registry-tls" not found
Jul 02 13:46:05.600380 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.600015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3d9299e-86c3-4cf4-9464-6a3cb72e0287-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-v7nvl\" (UID: \"e3d9299e-86c3-4cf4-9464-6a3cb72e0287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.600380 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:05.600054    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls podName:7d1807cd-b1d5-4fdc-af58-5212bb8ae739 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:06.100038335 +0000 UTC m=+65.941675827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls") pod "image-registry-84fb55f8df-spq4p" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739") : secret "image-registry-tls" not found
Jul 02 13:46:05.600380 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.600075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-trusted-ca\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.600380 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.600248    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef9709a9-9047-412d-a162-071bb4647e73-config\") pod \"service-ca-operator-5d7b7dc95c-6wr2h\" (UID: \"ef9709a9-9047-412d-a162-071bb4647e73\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.600380 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.600366    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-certificates\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.600816 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.600803    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9a4f6553-c51e-49aa-b0b5-dee8541cee26-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:05.601195 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.601172    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-trusted-ca\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.602425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.602405    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef9709a9-9047-412d-a162-071bb4647e73-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-6wr2h\" (UID: \"ef9709a9-9047-412d-a162-071bb4647e73\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.602735 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.602715    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-image-registry-private-configuration\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.602840 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.602822    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-installation-pull-secrets\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.611241 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.611219    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rsj7\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-kube-api-access-6rsj7\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.612221 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.612201    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grps\" (UniqueName: \"kubernetes.io/projected/ef9709a9-9047-412d-a162-071bb4647e73-kube-api-access-7grps\") pod \"service-ca-operator-5d7b7dc95c-6wr2h\" (UID: \"ef9709a9-9047-412d-a162-071bb4647e73\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.612785 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.612769    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-bound-sa-token\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:05.613036 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.613016    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bb2x\" (UniqueName: \"kubernetes.io/projected/9a4f6553-c51e-49aa-b0b5-dee8541cee26-kube-api-access-6bb2x\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:05.689789 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.689760    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9"]
Jul 02 13:46:05.692617 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:05.692592    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932531fc_a0e6_4e3a_a5ea_46a1357a6409.slice/crio-35c03d6fa0d737946a19b711ec9eabf85ea802af6ac868639cc29c566cd36ff3 WatchSource:0}: Error finding container 35c03d6fa0d737946a19b711ec9eabf85ea802af6ac868639cc29c566cd36ff3: Status 404 returned error can't find the container with id 35c03d6fa0d737946a19b711ec9eabf85ea802af6ac868639cc29c566cd36ff3
Jul 02 13:46:05.700848 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.700828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3d9299e-86c3-4cf4-9464-6a3cb72e0287-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-v7nvl\" (UID: \"e3d9299e-86c3-4cf4-9464-6a3cb72e0287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.700909 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.700862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjrm\" (UniqueName: \"kubernetes.io/projected/e3d9299e-86c3-4cf4-9464-6a3cb72e0287-kube-api-access-mzjrm\") pod \"kube-storage-version-migrator-operator-79964ff6f9-v7nvl\" (UID: \"e3d9299e-86c3-4cf4-9464-6a3cb72e0287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.700953 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.700914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pvhg\" (UniqueName: \"kubernetes.io/projected/32e69962-a074-45d3-8ab6-85bf35ad9801-kube-api-access-7pvhg\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:05.701133 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.701111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3d9299e-86c3-4cf4-9464-6a3cb72e0287-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-v7nvl\" (UID: \"e3d9299e-86c3-4cf4-9464-6a3cb72e0287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.701182 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.701166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:05.701315 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:05.701292    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:46:05.701404 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:05.701376    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls podName:32e69962-a074-45d3-8ab6-85bf35ad9801 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:06.20135395 +0000 UTC m=+66.042991451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cxtvr" (UID: "32e69962-a074-45d3-8ab6-85bf35ad9801") : secret "samples-operator-tls" not found
Jul 02 13:46:05.701654 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.701638    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3d9299e-86c3-4cf4-9464-6a3cb72e0287-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-v7nvl\" (UID: \"e3d9299e-86c3-4cf4-9464-6a3cb72e0287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.702913 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.702897    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3d9299e-86c3-4cf4-9464-6a3cb72e0287-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-v7nvl\" (UID: \"e3d9299e-86c3-4cf4-9464-6a3cb72e0287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.709200 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.709179    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pvhg\" (UniqueName: \"kubernetes.io/projected/32e69962-a074-45d3-8ab6-85bf35ad9801-kube-api-access-7pvhg\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:05.709576 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.709562    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjrm\" (UniqueName: \"kubernetes.io/projected/e3d9299e-86c3-4cf4-9464-6a3cb72e0287-kube-api-access-mzjrm\") pod \"kube-storage-version-migrator-operator-79964ff6f9-v7nvl\" (UID: \"e3d9299e-86c3-4cf4-9464-6a3cb72e0287\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.750153 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.750127    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9" event={"ID":"932531fc-a0e6-4e3a-a5ea-46a1357a6409","Type":"ContainerStarted","Data":"35c03d6fa0d737946a19b711ec9eabf85ea802af6ac868639cc29c566cd36ff3"}
Jul 02 13:46:05.777452 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.777437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"
Jul 02 13:46:05.887141 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.887108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"
Jul 02 13:46:05.896430 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:05.896406    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h"]
Jul 02 13:46:05.899048 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:05.899025    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9709a9_9047_412d_a162_071bb4647e73.slice/crio-ef0651e84ff76aa4ad4d8e31f3e10bf74cdc1c522256bd1903db81024422ef9a WatchSource:0}: Error finding container ef0651e84ff76aa4ad4d8e31f3e10bf74cdc1c522256bd1903db81024422ef9a: Status 404 returned error can't find the container with id ef0651e84ff76aa4ad4d8e31f3e10bf74cdc1c522256bd1903db81024422ef9a
Jul 02 13:46:06.008080 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:06.008053    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl"]
Jul 02 13:46:06.012143 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:06.012121    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d9299e_86c3_4cf4_9464_6a3cb72e0287.slice/crio-58cc8de853eff7e84e0588ad0cd1720a2f116f39d2b86ec9212a11a61e9a8917 WatchSource:0}: Error finding container 58cc8de853eff7e84e0588ad0cd1720a2f116f39d2b86ec9212a11a61e9a8917: Status 404 returned error can't find the container with id 58cc8de853eff7e84e0588ad0cd1720a2f116f39d2b86ec9212a11a61e9a8917
Jul 02 13:46:06.105690 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:06.105593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:06.105831 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:06.105727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:06.105831 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:06.105742    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:06.105831 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:06.105806    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls podName:9a4f6553-c51e-49aa-b0b5-dee8541cee26 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:07.105787588 +0000 UTC m=+66.947425083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-w5g2b" (UID: "9a4f6553-c51e-49aa-b0b5-dee8541cee26") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:06.106004 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:06.105846    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:46:06.106004 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:06.105861    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fb55f8df-spq4p: secret "image-registry-tls" not found
Jul 02 13:46:06.106004 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:06.105916    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls podName:7d1807cd-b1d5-4fdc-af58-5212bb8ae739 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:07.105898495 +0000 UTC m=+66.947535989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls") pod "image-registry-84fb55f8df-spq4p" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739") : secret "image-registry-tls" not found
Jul 02 13:46:06.206143 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:06.206108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:06.206297 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:06.206217    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:46:06.206297 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:06.206270    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls podName:32e69962-a074-45d3-8ab6-85bf35ad9801 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:07.206255473 +0000 UTC m=+67.047892965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cxtvr" (UID: "32e69962-a074-45d3-8ab6-85bf35ad9801") : secret "samples-operator-tls" not found
Jul 02 13:46:06.754112 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:06.754072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl" event={"ID":"e3d9299e-86c3-4cf4-9464-6a3cb72e0287","Type":"ContainerStarted","Data":"58cc8de853eff7e84e0588ad0cd1720a2f116f39d2b86ec9212a11a61e9a8917"}
Jul 02 13:46:06.755350 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:06.755319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h" event={"ID":"ef9709a9-9047-412d-a162-071bb4647e73","Type":"ContainerStarted","Data":"ef0651e84ff76aa4ad4d8e31f3e10bf74cdc1c522256bd1903db81024422ef9a"}
Jul 02 13:46:07.114404 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:07.114363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:07.114626 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:07.114438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:07.114626 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:07.114545    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:46:07.114626 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:07.114567    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fb55f8df-spq4p: secret "image-registry-tls" not found
Jul 02 13:46:07.114626 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:07.114595    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:07.114911 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:07.114649    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls podName:7d1807cd-b1d5-4fdc-af58-5212bb8ae739 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:09.114627215 +0000 UTC m=+68.956264722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls") pod "image-registry-84fb55f8df-spq4p" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739") : secret "image-registry-tls" not found
Jul 02 13:46:07.114911 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:07.114667    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls podName:9a4f6553-c51e-49aa-b0b5-dee8541cee26 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:09.114658804 +0000 UTC m=+68.956296299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-w5g2b" (UID: "9a4f6553-c51e-49aa-b0b5-dee8541cee26") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:07.215104 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:07.215068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:07.215268 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:07.215219    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:46:07.215314 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:07.215308    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls podName:32e69962-a074-45d3-8ab6-85bf35ad9801 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:09.215293862 +0000 UTC m=+69.056931354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cxtvr" (UID: "32e69962-a074-45d3-8ab6-85bf35ad9801") : secret "samples-operator-tls" not found
Jul 02 13:46:07.758711 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:07.758679    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9" event={"ID":"932531fc-a0e6-4e3a-a5ea-46a1357a6409","Type":"ContainerStarted","Data":"26ea6c9acdd099a5ffb618d92ee117aebeab6cf19b266502dc0cdd8db594730f"}
Jul 02 13:46:07.774115 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:07.774067    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cl6t9" podStartSLOduration=1.513287984 podStartE2EDuration="2.774050816s" totalImagesPullingTime="1.260762832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:07.773939353 +0000 UTC m=+67.615576869" watchObservedRunningTime="2026-07-02 13:46:07.774050816 +0000 UTC m=+67.615688331"
Jul 02 13:46:08.021528 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.021424    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:46:08.021755 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.021568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:46:08.024346 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.024323    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:46:08.024633 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.024607    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:46:08.032524 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:08.032499    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 02 13:46:08.032619 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:08.032576    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs podName:66cf39f9-4496-49ad-aabc-e9453dab385a nodeName:}" failed. No retries permitted until 2026-07-02 13:47:12.03255373 +0000 UTC m=+131.874191225 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs") pod "network-metrics-daemon-dm4g2" (UID: "66cf39f9-4496-49ad-aabc-e9453dab385a") : secret "metrics-daemon-secret" not found
Jul 02 13:46:08.035198 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.035173    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ad7ad64-df1a-46f9-8e19-2b7333b838f5-original-pull-secret\") pod \"global-pull-secret-syncer-5mxmz\" (UID: \"4ad7ad64-df1a-46f9-8e19-2b7333b838f5\") " pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:46:08.116699 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.116666    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mxmz"
Jul 02 13:46:08.122895 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.122871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:46:08.125888 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.125864    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:46:08.136193 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.136165    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:46:08.146964 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.146940    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fmb\" (UniqueName: \"kubernetes.io/projected/d5ada156-013a-43b9-a444-d4fd7984a5be-kube-api-access-m7fmb\") pod \"network-check-target-zn2d9\" (UID: \"d5ada156-013a-43b9-a444-d4fd7984a5be\") " pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:46:08.412856 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.412777    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nskqf\""
Jul 02 13:46:08.420542 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.420519    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:46:08.946621 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.946583    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zn2d9"]
Jul 02 13:46:08.950426 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:08.950401    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ada156_013a_43b9_a444_d4fd7984a5be.slice/crio-2b1c4d386ba4332b0e427e66b27e53e5e796881681022d57e347659abc5fa147 WatchSource:0}: Error finding container 2b1c4d386ba4332b0e427e66b27e53e5e796881681022d57e347659abc5fa147: Status 404 returned error can't find the container with id 2b1c4d386ba4332b0e427e66b27e53e5e796881681022d57e347659abc5fa147
Jul 02 13:46:08.957130 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:08.957111    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5mxmz"]
Jul 02 13:46:08.960354 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:08.960333    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad7ad64_df1a_46f9_8e19_2b7333b838f5.slice/crio-2ca2c25f9068fca4312845a0e7fe5c3c6bfd505c887d10cd162f68f9c386a00a WatchSource:0}: Error finding container 2ca2c25f9068fca4312845a0e7fe5c3c6bfd505c887d10cd162f68f9c386a00a: Status 404 returned error can't find the container with id 2ca2c25f9068fca4312845a0e7fe5c3c6bfd505c887d10cd162f68f9c386a00a
Jul 02 13:46:09.133358 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:09.133251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:09.133358 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:09.133315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:09.133682 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:09.133406    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:46:09.133682 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:09.133423    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fb55f8df-spq4p: secret "image-registry-tls" not found
Jul 02 13:46:09.133682 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:09.133432    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:09.133682 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:09.133499    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls podName:7d1807cd-b1d5-4fdc-af58-5212bb8ae739 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:13.13347038 +0000 UTC m=+72.975107888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls") pod "image-registry-84fb55f8df-spq4p" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739") : secret "image-registry-tls" not found
Jul 02 13:46:09.133682 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:09.133516    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls podName:9a4f6553-c51e-49aa-b0b5-dee8541cee26 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:13.133508325 +0000 UTC m=+72.975145816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-w5g2b" (UID: "9a4f6553-c51e-49aa-b0b5-dee8541cee26") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:09.234073 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:09.234032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:09.234258 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:09.234152    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:46:09.234258 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:09.234209    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls podName:32e69962-a074-45d3-8ab6-85bf35ad9801 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:13.234194583 +0000 UTC m=+73.075832075 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cxtvr" (UID: "32e69962-a074-45d3-8ab6-85bf35ad9801") : secret "samples-operator-tls" not found
Jul 02 13:46:09.435782 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:09.435751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:46:09.435994 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:09.435977    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:46:09.436057 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:09.436046    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls podName:aedca59b-0e89-4e5b-9092-3628bfc55297 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:41.436027491 +0000 UTC m=+101.277664983 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls") pod "dns-default-qkjj8" (UID: "aedca59b-0e89-4e5b-9092-3628bfc55297") : secret "dns-default-metrics-tls" not found
Jul 02 13:46:09.767102 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:09.766985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5mxmz" event={"ID":"4ad7ad64-df1a-46f9-8e19-2b7333b838f5","Type":"ContainerStarted","Data":"2ca2c25f9068fca4312845a0e7fe5c3c6bfd505c887d10cd162f68f9c386a00a"}
Jul 02 13:46:09.769110 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:09.768930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl" event={"ID":"e3d9299e-86c3-4cf4-9464-6a3cb72e0287","Type":"ContainerStarted","Data":"c3fc37209edde8c2269a75599594e3acb00daba3a54f1754ca65193540c5c4e9"}
Jul 02 13:46:09.771887 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:09.771857    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h" event={"ID":"ef9709a9-9047-412d-a162-071bb4647e73","Type":"ContainerStarted","Data":"74c5e84ed41463a92b2f5dd0a304e77d453b7e5cd462dc677c01795f58bcd0fd"}
Jul 02 13:46:09.773651 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:09.773622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zn2d9" event={"ID":"d5ada156-013a-43b9-a444-d4fd7984a5be","Type":"ContainerStarted","Data":"2b1c4d386ba4332b0e427e66b27e53e5e796881681022d57e347659abc5fa147"}
Jul 02 13:46:09.788350 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:09.788259    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl" podStartSLOduration=1.985520314 podStartE2EDuration="4.788240921s" totalImagesPullingTime="2.802720607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:09.786969298 +0000 UTC m=+69.628606807" watchObservedRunningTime="2026-07-02 13:46:09.788240921 +0000 UTC m=+69.629878418"
Jul 02 13:46:11.112241 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.112187    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h" podStartSLOduration=3.204320581 podStartE2EDuration="6.112168586s" totalImagesPullingTime="2.907848005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:09.805732756 +0000 UTC m=+69.647370274" watchObservedRunningTime="2026-07-02 13:46:11.112168586 +0000 UTC m=+70.953806099"
Jul 02 13:46:11.112703 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.112329    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jshgg"]
Jul 02 13:46:11.141537 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.141507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jshgg"]
Jul 02 13:46:11.141700 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.141614    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.145923 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.145889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 13:46:11.146619 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.146553    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 13:46:11.146767 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.146751    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 13:46:11.148208 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.148147    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 13:46:11.149095 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.149070    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvtkr\""
Jul 02 13:46:11.254026 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.253954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1d430fe-cf31-42f5-b780-28352f84d9dd-data-volume\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.254026 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.254012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.254262 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.254064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1d430fe-cf31-42f5-b780-28352f84d9dd-crio-socket\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.254262 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.254092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvdfs\" (UniqueName: \"kubernetes.io/projected/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-api-access-zvdfs\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.254362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.254267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.355714 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.355673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1d430fe-cf31-42f5-b780-28352f84d9dd-data-volume\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.355863 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.355725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.355863 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.355761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1d430fe-cf31-42f5-b780-28352f84d9dd-crio-socket\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.355863 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.355831    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1d430fe-cf31-42f5-b780-28352f84d9dd-crio-socket\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.356002 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.355871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvdfs\" (UniqueName: \"kubernetes.io/projected/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-api-access-zvdfs\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.356002 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:11.355885    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:46:11.356002 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.355937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.356002 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:11.355977    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:11.855955609 +0000 UTC m=+71.697593104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : configmap "kube-rbac-proxy" not found
Jul 02 13:46:11.356186 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:11.356030    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:11.356186 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:11.356079    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:11.856062828 +0000 UTC m=+71.697700334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:11.356186 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.356129    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1d430fe-cf31-42f5-b780-28352f84d9dd-data-volume\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.365259 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.365072    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvdfs\" (UniqueName: \"kubernetes.io/projected/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-api-access-zvdfs\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.860222 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.860176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.860405 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:11.860288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:11.860459 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:11.860428    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:46:11.860541 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:11.860524    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:12.860500843 +0000 UTC m=+72.702138341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : configmap "kube-rbac-proxy" not found
Jul 02 13:46:11.860676 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:11.860641    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:11.860741 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:11.860727    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:12.860711096 +0000 UTC m=+72.702348603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:12.384071 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.384032    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"]
Jul 02 13:46:12.386441 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.386421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.390062 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.390040    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bqjmb\""
Jul 02 13:46:12.391059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.391037    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 02 13:46:12.391130 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.391107    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 02 13:46:12.391198 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.391107    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 02 13:46:12.391976 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.391960    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 02 13:46:12.404718 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.404690    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"]
Jul 02 13:46:12.465399 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.465360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0c80df0-87d8-4534-8278-a3592acac910-signing-key\") pod \"service-ca-7bb4f4bcc7-9h46f\" (UID: \"e0c80df0-87d8-4534-8278-a3592acac910\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.465588 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.465542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0c80df0-87d8-4534-8278-a3592acac910-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-9h46f\" (UID: \"e0c80df0-87d8-4534-8278-a3592acac910\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.465661 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.465640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b54d\" (UniqueName: \"kubernetes.io/projected/e0c80df0-87d8-4534-8278-a3592acac910-kube-api-access-2b54d\") pod \"service-ca-7bb4f4bcc7-9h46f\" (UID: \"e0c80df0-87d8-4534-8278-a3592acac910\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.566652 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.566617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0c80df0-87d8-4534-8278-a3592acac910-signing-key\") pod \"service-ca-7bb4f4bcc7-9h46f\" (UID: \"e0c80df0-87d8-4534-8278-a3592acac910\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.566837 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.566721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0c80df0-87d8-4534-8278-a3592acac910-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-9h46f\" (UID: \"e0c80df0-87d8-4534-8278-a3592acac910\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.566837 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.566790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b54d\" (UniqueName: \"kubernetes.io/projected/e0c80df0-87d8-4534-8278-a3592acac910-kube-api-access-2b54d\") pod \"service-ca-7bb4f4bcc7-9h46f\" (UID: \"e0c80df0-87d8-4534-8278-a3592acac910\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.567518 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.567470    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0c80df0-87d8-4534-8278-a3592acac910-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-9h46f\" (UID: \"e0c80df0-87d8-4534-8278-a3592acac910\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.569315 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.569293    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0c80df0-87d8-4534-8278-a3592acac910-signing-key\") pod \"service-ca-7bb4f4bcc7-9h46f\" (UID: \"e0c80df0-87d8-4534-8278-a3592acac910\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.577705 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.577681    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b54d\" (UniqueName: \"kubernetes.io/projected/e0c80df0-87d8-4534-8278-a3592acac910-kube-api-access-2b54d\") pod \"service-ca-7bb4f4bcc7-9h46f\" (UID: \"e0c80df0-87d8-4534-8278-a3592acac910\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.697540 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.697478    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"
Jul 02 13:46:12.869851 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.869808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:12.870046 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:12.869935    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:46:12.870046 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:12.870004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:12.870046 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:12.870025    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:14.870004781 +0000 UTC m=+74.711642279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : configmap "kube-rbac-proxy" not found
Jul 02 13:46:12.870216 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:12.870105    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:12.870216 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:12.870160    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:14.870144719 +0000 UTC m=+74.711782215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:13.172449 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.172355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:13.172449 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.172441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:13.172661 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:13.172555    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:13.172661 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:13.172603    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:46:13.172661 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:13.172622    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fb55f8df-spq4p: secret "image-registry-tls" not found
Jul 02 13:46:13.172661 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:13.172634    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls podName:9a4f6553-c51e-49aa-b0b5-dee8541cee26 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:21.172613339 +0000 UTC m=+81.014250833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-w5g2b" (UID: "9a4f6553-c51e-49aa-b0b5-dee8541cee26") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:13.172845 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:13.172688    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls podName:7d1807cd-b1d5-4fdc-af58-5212bb8ae739 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:21.172669947 +0000 UTC m=+81.014307441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls") pod "image-registry-84fb55f8df-spq4p" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739") : secret "image-registry-tls" not found
Jul 02 13:46:13.273080 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.273043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:13.273246 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:13.273203    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:46:13.273292 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:13.273280    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls podName:32e69962-a074-45d3-8ab6-85bf35ad9801 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:21.273264225 +0000 UTC m=+81.114901717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cxtvr" (UID: "32e69962-a074-45d3-8ab6-85bf35ad9801") : secret "samples-operator-tls" not found
Jul 02 13:46:13.643052 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.643021    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-9h46f"]
Jul 02 13:46:13.646638 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:13.646608    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c80df0_87d8_4534_8278_a3592acac910.slice/crio-f65edbb688526a1c487e26117ec586226621a7ed1bdf18f29c053ceb6824c7d7 WatchSource:0}: Error finding container f65edbb688526a1c487e26117ec586226621a7ed1bdf18f29c053ceb6824c7d7: Status 404 returned error can't find the container with id f65edbb688526a1c487e26117ec586226621a7ed1bdf18f29c053ceb6824c7d7
Jul 02 13:46:13.790524 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.790415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zn2d9" event={"ID":"d5ada156-013a-43b9-a444-d4fd7984a5be","Type":"ContainerStarted","Data":"e40c5f7e8bc2b8cffc9302f44ff9e8d23ca53dd1d5a053811f111ba820bd7283"}
Jul 02 13:46:13.790690 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.790537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:46:13.792106 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.792074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f" event={"ID":"e0c80df0-87d8-4534-8278-a3592acac910","Type":"ContainerStarted","Data":"14348e77cb5fc2c93bde1b784b4487ea2711fef1bf4a20fe0ee8ebdabcbc3fa5"}
Jul 02 13:46:13.792236 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.792121    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f" event={"ID":"e0c80df0-87d8-4534-8278-a3592acac910","Type":"ContainerStarted","Data":"f65edbb688526a1c487e26117ec586226621a7ed1bdf18f29c053ceb6824c7d7"}
Jul 02 13:46:13.793423 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.793395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5mxmz" event={"ID":"4ad7ad64-df1a-46f9-8e19-2b7333b838f5","Type":"ContainerStarted","Data":"284ecc2974b0659d0127ebd9a743e787fecf32fc9551d14a02b5cf8023ddd62b"}
Jul 02 13:46:13.843276 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.843233    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zn2d9" podStartSLOduration=66.290146267 podStartE2EDuration="1m10.843218153s" totalImagesPullingTime="4.553071886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:13.81291501 +0000 UTC m=+73.654552525" watchObservedRunningTime="2026-07-02 13:46:13.843218153 +0000 UTC m=+73.684855663"
Jul 02 13:46:13.867579 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:13.867540    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-9h46f" podStartSLOduration=1.867527558 podStartE2EDuration="1.867527558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:13.843204322 +0000 UTC m=+73.684841835" watchObservedRunningTime="2026-07-02 13:46:13.867527558 +0000 UTC m=+73.709165071"
Jul 02 13:46:14.120716 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:14.120654    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rq799_2a804050-c3a7-4cf3-923b-5146f3dde7c3/dns-node-resolver/0.log"
Jul 02 13:46:14.886835 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:14.886800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:14.887260 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:14.886938    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:14.887260 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:14.887004    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:18.886983522 +0000 UTC m=+78.728621034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:14.887260 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:14.887061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:14.887260 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:14.887185    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:46:14.887260 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:14.887223    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:18.88721135 +0000 UTC m=+78.728848846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : configmap "kube-rbac-proxy" not found
Jul 02 13:46:15.117628 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:15.117601    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnjj7_095b3c78-4147-4098-b517-722b1e4941f1/node-ca/0.log"
Jul 02 13:46:15.519435 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:15.519383    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2ljkk_8eaabca2-d6be-46b6-9b5d-d7db2490f413/serve-healthcheck-canary/0.log"
Jul 02 13:46:16.519410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:16.519377    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-v7nvl_e3d9299e-86c3-4cf4-9464-6a3cb72e0287/kube-storage-version-migrator-operator/0.log"
Jul 02 13:46:18.918358 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:18.918319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:18.918744 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:18.918401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:18.918744 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:18.918459    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:18.918744 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:18.918536    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:26.918521473 +0000 UTC m=+86.760158968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:46:18.918744 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:18.918534    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:46:18.918744 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:18.918564    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm podName:a1d430fe-cf31-42f5-b780-28352f84d9dd nodeName:}" failed. No retries permitted until 2026-07-02 13:46:26.918556871 +0000 UTC m=+86.760194363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jshgg" (UID: "a1d430fe-cf31-42f5-b780-28352f84d9dd") : configmap "kube-rbac-proxy" not found
Jul 02 13:46:21.234549 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.234513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:21.234921 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.234569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:21.234921 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:21.234664    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:21.234921 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:21.234725    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls podName:9a4f6553-c51e-49aa-b0b5-dee8541cee26 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:37.234710626 +0000 UTC m=+97.076348118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-w5g2b" (UID: "9a4f6553-c51e-49aa-b0b5-dee8541cee26") : secret "cluster-monitoring-operator-tls" not found
Jul 02 13:46:21.237109 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.237066    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") pod \"image-registry-84fb55f8df-spq4p\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") " pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:21.281947 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.281924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:21.335198 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.335166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:21.337769 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.337628    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32e69962-a074-45d3-8ab6-85bf35ad9801-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cxtvr\" (UID: \"32e69962-a074-45d3-8ab6-85bf35ad9801\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:21.395880 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.395857    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"
Jul 02 13:46:21.410432 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.410381    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5mxmz" podStartSLOduration=73.860932156 podStartE2EDuration="1m18.410363731s" totalImagesPullingTime="4.549431575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:13.867090831 +0000 UTC m=+73.708728346" watchObservedRunningTime="2026-07-02 13:46:21.410363731 +0000 UTC m=+81.252001249"
Jul 02 13:46:21.410562 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.410545    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84fb55f8df-spq4p"]
Jul 02 13:46:21.413790 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:21.413765    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1807cd_b1d5_4fdc_af58_5212bb8ae739.slice/crio-9d1bdad1f8e7bc241ebedaed526cf6a1e702cd81a3305fb1b31c5d0c047b6987 WatchSource:0}: Error finding container 9d1bdad1f8e7bc241ebedaed526cf6a1e702cd81a3305fb1b31c5d0c047b6987: Status 404 returned error can't find the container with id 9d1bdad1f8e7bc241ebedaed526cf6a1e702cd81a3305fb1b31c5d0c047b6987
Jul 02 13:46:21.522709 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.522653    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr"]
Jul 02 13:46:21.825656 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.825574    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr" event={"ID":"32e69962-a074-45d3-8ab6-85bf35ad9801","Type":"ContainerStarted","Data":"a94194dd66bfa0b0dd4bb912b4b00ddfe334753da2828bd079e117cc26b1ff94"}
Jul 02 13:46:21.827085 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.827054    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p" event={"ID":"7d1807cd-b1d5-4fdc-af58-5212bb8ae739","Type":"ContainerStarted","Data":"fe1ecd97144e2f935e06ee12df371785e61a7eb6cf10483ad7079723be972787"}
Jul 02 13:46:21.827205 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.827097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p" event={"ID":"7d1807cd-b1d5-4fdc-af58-5212bb8ae739","Type":"ContainerStarted","Data":"9d1bdad1f8e7bc241ebedaed526cf6a1e702cd81a3305fb1b31c5d0c047b6987"}
Jul 02 13:46:21.827205 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.827137    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:21.847018 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:21.846972    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p" podStartSLOduration=16.846957571 podStartE2EDuration="16.846957571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:21.845797709 +0000 UTC m=+81.687435228" watchObservedRunningTime="2026-07-02 13:46:21.846957571 +0000 UTC m=+81.688595085"
Jul 02 13:46:23.835093 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:23.835058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr" event={"ID":"32e69962-a074-45d3-8ab6-85bf35ad9801","Type":"ContainerStarted","Data":"908e03412661564be608d2de2ba4b0ccfbbf593f88a857f712225803063c8934"}
Jul 02 13:46:23.835419 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:23.835113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr" event={"ID":"32e69962-a074-45d3-8ab6-85bf35ad9801","Type":"ContainerStarted","Data":"9e52869e17a173aba5e7e6db31a198075f659f5abc06d591104f5d11eb9a55c5"}
Jul 02 13:46:23.857929 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:23.857882    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cxtvr" podStartSLOduration=16.786328825 podStartE2EDuration="18.857868073s" totalImagesPullingTime="2.071539248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:23.857034056 +0000 UTC m=+83.698671573" watchObservedRunningTime="2026-07-02 13:46:23.857868073 +0000 UTC m=+83.699505586"
Jul 02 13:46:26.980270 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:26.980237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:26.980836 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:26.980296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:26.980836 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:26.980758    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1d430fe-cf31-42f5-b780-28352f84d9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:26.982592 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:26.982568    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1d430fe-cf31-42f5-b780-28352f84d9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jshgg\" (UID: \"a1d430fe-cf31-42f5-b780-28352f84d9dd\") " pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:27.053010 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:27.052989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jshgg"
Jul 02 13:46:27.179626 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:27.179595    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jshgg"]
Jul 02 13:46:27.182302 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:27.182274    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d430fe_cf31_42f5_b780_28352f84d9dd.slice/crio-1bc961320233656b8fcf9607c9a0aed7aa4e3dd41a27ad9eb02f696c10cfe701 WatchSource:0}: Error finding container 1bc961320233656b8fcf9607c9a0aed7aa4e3dd41a27ad9eb02f696c10cfe701: Status 404 returned error can't find the container with id 1bc961320233656b8fcf9607c9a0aed7aa4e3dd41a27ad9eb02f696c10cfe701
Jul 02 13:46:27.850501 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:27.850442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jshgg" event={"ID":"a1d430fe-cf31-42f5-b780-28352f84d9dd","Type":"ContainerStarted","Data":"1bc961320233656b8fcf9607c9a0aed7aa4e3dd41a27ad9eb02f696c10cfe701"}
Jul 02 13:46:31.866386 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:31.866308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jshgg" event={"ID":"a1d430fe-cf31-42f5-b780-28352f84d9dd","Type":"ContainerStarted","Data":"a4e505e5cee32f0c6d03dfb92408b0c6de51dbaf85e094d70c1fc368f9fd4fa8"}
Jul 02 13:46:31.866386 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:31.866362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jshgg" event={"ID":"a1d430fe-cf31-42f5-b780-28352f84d9dd","Type":"ContainerStarted","Data":"396d1f8fc2016bf7c8d8e84c73530bf4753ebc3664f4d3b0908fac2e6055276c"}
Jul 02 13:46:34.876958 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:34.876925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jshgg" event={"ID":"a1d430fe-cf31-42f5-b780-28352f84d9dd","Type":"ContainerStarted","Data":"cc7138cba3654ea94ed322113edff4cb5547e3711e267217d596163cfdc7b954"}
Jul 02 13:46:34.894720 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:34.894676    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jshgg" podStartSLOduration=17.042140202 podStartE2EDuration="23.894649944s" totalImagesPullingTime="6.852509742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:46:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:34.894161986 +0000 UTC m=+94.735799502" watchObservedRunningTime="2026-07-02 13:46:34.894649944 +0000 UTC m=+94.736287452"
Jul 02 13:46:36.519685 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.519658    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"]
Jul 02 13:46:36.522690 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.522670    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84fb55f8df-spq4p"]
Jul 02 13:46:36.522850 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.522831    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"
Jul 02 13:46:36.525543 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.525525    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 13:46:36.525659 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.525611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8wx5f\""
Jul 02 13:46:36.525659 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.525633    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 13:46:36.527251 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.527227    2566 patch_prober.go:28] interesting pod/image-registry-84fb55f8df-spq4p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 02 13:46:36.527369 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.527276    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p" podUID="7d1807cd-b1d5-4fdc-af58-5212bb8ae739" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:46:36.532526 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.532502    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"]
Jul 02 13:46:36.655396 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.655369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cd8ea55-fb58-4a20-ad89-fbbb017846fd-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-wx95f\" (UID: \"5cd8ea55-fb58-4a20-ad89-fbbb017846fd\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"
Jul 02 13:46:36.655555 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.655429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cd8ea55-fb58-4a20-ad89-fbbb017846fd-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-wx95f\" (UID: \"5cd8ea55-fb58-4a20-ad89-fbbb017846fd\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"
Jul 02 13:46:36.689805 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.689782    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-n7nq7"]
Jul 02 13:46:36.692491 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.692465    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-n7nq7"
Jul 02 13:46:36.696200 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.696183    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xmsjx\""
Jul 02 13:46:36.696459 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.696444    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 13:46:36.698623 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.698604    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 13:46:36.721992 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.721972    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-n7nq7"]
Jul 02 13:46:36.756343 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.756324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cd8ea55-fb58-4a20-ad89-fbbb017846fd-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-wx95f\" (UID: \"5cd8ea55-fb58-4a20-ad89-fbbb017846fd\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"
Jul 02 13:46:36.756423 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.756379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cd8ea55-fb58-4a20-ad89-fbbb017846fd-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-wx95f\" (UID: \"5cd8ea55-fb58-4a20-ad89-fbbb017846fd\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"
Jul 02 13:46:36.756950 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.756927    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cd8ea55-fb58-4a20-ad89-fbbb017846fd-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-wx95f\" (UID: \"5cd8ea55-fb58-4a20-ad89-fbbb017846fd\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"
Jul 02 13:46:36.758627 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.758609    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cd8ea55-fb58-4a20-ad89-fbbb017846fd-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-wx95f\" (UID: \"5cd8ea55-fb58-4a20-ad89-fbbb017846fd\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"
Jul 02 13:46:36.833933 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.833881    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"
Jul 02 13:46:36.857821 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.857794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffts\" (UniqueName: \"kubernetes.io/projected/0a0dfc85-0d34-4338-a559-553b52c97cd2-kube-api-access-6ffts\") pod \"downloads-5499b6fb8f-n7nq7\" (UID: \"0a0dfc85-0d34-4338-a559-553b52c97cd2\") " pod="openshift-console/downloads-5499b6fb8f-n7nq7"
Jul 02 13:46:36.959117 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.959088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffts\" (UniqueName: \"kubernetes.io/projected/0a0dfc85-0d34-4338-a559-553b52c97cd2-kube-api-access-6ffts\") pod \"downloads-5499b6fb8f-n7nq7\" (UID: \"0a0dfc85-0d34-4338-a559-553b52c97cd2\") " pod="openshift-console/downloads-5499b6fb8f-n7nq7"
Jul 02 13:46:36.960809 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.960779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f"]
Jul 02 13:46:36.963828 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:36.963804    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd8ea55_fb58_4a20_ad89_fbbb017846fd.slice/crio-1cfddad4a9822d9fe9a3f935299863b807289c50cc0639386dc75e71de72ebaf WatchSource:0}: Error finding container 1cfddad4a9822d9fe9a3f935299863b807289c50cc0639386dc75e71de72ebaf: Status 404 returned error can't find the container with id 1cfddad4a9822d9fe9a3f935299863b807289c50cc0639386dc75e71de72ebaf
Jul 02 13:46:36.970644 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:36.970624    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffts\" (UniqueName: \"kubernetes.io/projected/0a0dfc85-0d34-4338-a559-553b52c97cd2-kube-api-access-6ffts\") pod \"downloads-5499b6fb8f-n7nq7\" (UID: \"0a0dfc85-0d34-4338-a559-553b52c97cd2\") " pod="openshift-console/downloads-5499b6fb8f-n7nq7"
Jul 02 13:46:37.001658 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:37.001634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-n7nq7"
Jul 02 13:46:37.152425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:37.152399    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-n7nq7"]
Jul 02 13:46:37.154623 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:37.154597    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0dfc85_0d34_4338_a559_553b52c97cd2.slice/crio-57a11e8c0743120b03bbf1ee2fd9804a9a17a8c9868d87657282ec838a7d1dbd WatchSource:0}: Error finding container 57a11e8c0743120b03bbf1ee2fd9804a9a17a8c9868d87657282ec838a7d1dbd: Status 404 returned error can't find the container with id 57a11e8c0743120b03bbf1ee2fd9804a9a17a8c9868d87657282ec838a7d1dbd
Jul 02 13:46:37.262924 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:37.262897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:37.265220 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:37.265200    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a4f6553-c51e-49aa-b0b5-dee8541cee26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-w5g2b\" (UID: \"9a4f6553-c51e-49aa-b0b5-dee8541cee26\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:37.487501 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:37.487459    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"
Jul 02 13:46:37.609876 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:37.609848    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b"]
Jul 02 13:46:37.612872 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:37.612848    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4f6553_c51e_49aa_b0b5_dee8541cee26.slice/crio-6545eb0154be777c87fa11b64e6e48754be9a91b382354c7e21403c7a955a0cb WatchSource:0}: Error finding container 6545eb0154be777c87fa11b64e6e48754be9a91b382354c7e21403c7a955a0cb: Status 404 returned error can't find the container with id 6545eb0154be777c87fa11b64e6e48754be9a91b382354c7e21403c7a955a0cb
Jul 02 13:46:37.887239 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:37.887163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b" event={"ID":"9a4f6553-c51e-49aa-b0b5-dee8541cee26","Type":"ContainerStarted","Data":"6545eb0154be777c87fa11b64e6e48754be9a91b382354c7e21403c7a955a0cb"}
Jul 02 13:46:37.888402 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:37.888368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-n7nq7" event={"ID":"0a0dfc85-0d34-4338-a559-553b52c97cd2","Type":"ContainerStarted","Data":"57a11e8c0743120b03bbf1ee2fd9804a9a17a8c9868d87657282ec838a7d1dbd"}
Jul 02 13:46:37.889540 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:37.889515    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f" event={"ID":"5cd8ea55-fb58-4a20-ad89-fbbb017846fd","Type":"ContainerStarted","Data":"1cfddad4a9822d9fe9a3f935299863b807289c50cc0639386dc75e71de72ebaf"}
Jul 02 13:46:38.895263 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:38.895215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f" event={"ID":"5cd8ea55-fb58-4a20-ad89-fbbb017846fd","Type":"ContainerStarted","Data":"7d28a17384379f062e2c34988cc9dec81256bf525e24fd71c63c2fdc605bac27"}
Jul 02 13:46:38.912154 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:38.912098    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-wx95f" podStartSLOduration=1.5945752359999998 podStartE2EDuration="2.912078189s" totalImagesPullingTime="1.317502953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:38.910248131 +0000 UTC m=+98.751885648" watchObservedRunningTime="2026-07-02 13:46:38.912078189 +0000 UTC m=+98.753715704"
Jul 02 13:46:39.579573 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.579527    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74dc7dbb84-grc9m"]
Jul 02 13:46:39.584406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.584378    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.587891 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.587866    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h2hd7\""
Jul 02 13:46:39.587891 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.587885    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 13:46:39.588084 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.587891    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 13:46:39.588195 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.588168    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 13:46:39.588309 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.588240    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 13:46:39.588662 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.588642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 13:46:39.602127 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.602072    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dc7dbb84-grc9m"]
Jul 02 13:46:39.682064 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.682010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5hw\" (UniqueName: \"kubernetes.io/projected/018d4c9d-e729-41f4-8d8d-3395172787bd-kube-api-access-ts5hw\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.682217 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.682083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-oauth-serving-cert\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.682217 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.682126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-console-config\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.682217 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.682156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-serving-cert\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.682431 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.682230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-service-ca\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.682431 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.682270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-oauth-config\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.783275 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.783237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-service-ca\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.783449 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.783290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-oauth-config\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.783449 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.783326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5hw\" (UniqueName: \"kubernetes.io/projected/018d4c9d-e729-41f4-8d8d-3395172787bd-kube-api-access-ts5hw\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.783449 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.783370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-oauth-serving-cert\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.783449 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.783413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-console-config\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.783449 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.783442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-serving-cert\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.784091 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.784049    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-service-ca\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.784192 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.784132    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-oauth-serving-cert\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.784315 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.784257    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-console-config\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.786367 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.786340    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-oauth-config\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.786547 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.786478    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-serving-cert\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.793498 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.793461    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5hw\" (UniqueName: \"kubernetes.io/projected/018d4c9d-e729-41f4-8d8d-3395172787bd-kube-api-access-ts5hw\") pod \"console-74dc7dbb84-grc9m\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") " pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:39.896734 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:39.896703    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:40.047222 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.047184    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dc7dbb84-grc9m"]
Jul 02 13:46:40.050827 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:40.050795    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018d4c9d_e729_41f4_8d8d_3395172787bd.slice/crio-683f7567c771c2f3a73601f7a6f10301a1493afa74285e39b15d302470064576 WatchSource:0}: Error finding container 683f7567c771c2f3a73601f7a6f10301a1493afa74285e39b15d302470064576: Status 404 returned error can't find the container with id 683f7567c771c2f3a73601f7a6f10301a1493afa74285e39b15d302470064576
Jul 02 13:46:40.382213 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.382178    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"]
Jul 02 13:46:40.385630 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.385609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"
Jul 02 13:46:40.390739 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.390718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 02 13:46:40.391147 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.391124    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-45r7v\""
Jul 02 13:46:40.406689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.406667    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"]
Jul 02 13:46:40.489425 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.489394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0b6446cb-36a3-44d4-a51e-21547a9186c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-v48zk\" (UID: \"0b6446cb-36a3-44d4-a51e-21547a9186c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"
Jul 02 13:46:40.590657 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.590182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0b6446cb-36a3-44d4-a51e-21547a9186c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-v48zk\" (UID: \"0b6446cb-36a3-44d4-a51e-21547a9186c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"
Jul 02 13:46:40.590657 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:40.590311    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 13:46:40.590657 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:40.590377    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6446cb-36a3-44d4-a51e-21547a9186c0-tls-certificates podName:0b6446cb-36a3-44d4-a51e-21547a9186c0 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:41.0903588 +0000 UTC m=+100.931996300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/0b6446cb-36a3-44d4-a51e-21547a9186c0-tls-certificates") pod "prometheus-operator-admission-webhook-7ddc8784d4-v48zk" (UID: "0b6446cb-36a3-44d4-a51e-21547a9186c0") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 02 13:46:40.909225 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.909184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b" event={"ID":"9a4f6553-c51e-49aa-b0b5-dee8541cee26","Type":"ContainerStarted","Data":"44f94a006459c7b2bbc2fbb2878a6aca46a5ddee460953603450dac5b1622a9d"}
Jul 02 13:46:40.910582 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.910543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dc7dbb84-grc9m" event={"ID":"018d4c9d-e729-41f4-8d8d-3395172787bd","Type":"ContainerStarted","Data":"683f7567c771c2f3a73601f7a6f10301a1493afa74285e39b15d302470064576"}
Jul 02 13:46:40.927909 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:40.927865    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-w5g2b" podStartSLOduration=33.689261382 podStartE2EDuration="35.927845523s" totalImagesPullingTime="2.238584141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:40.926677851 +0000 UTC m=+100.768315405" watchObservedRunningTime="2026-07-02 13:46:40.927845523 +0000 UTC m=+100.769483029"
Jul 02 13:46:41.094591 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.094552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0b6446cb-36a3-44d4-a51e-21547a9186c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-v48zk\" (UID: \"0b6446cb-36a3-44d4-a51e-21547a9186c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"
Jul 02 13:46:41.097630 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.097603    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0b6446cb-36a3-44d4-a51e-21547a9186c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-v48zk\" (UID: \"0b6446cb-36a3-44d4-a51e-21547a9186c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"
Jul 02 13:46:41.296548 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.296512    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"
Jul 02 13:46:41.452267 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.452223    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"]
Jul 02 13:46:41.457465 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:41.457411    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6446cb_36a3_44d4_a51e_21547a9186c0.slice/crio-8e9f126308e046f45340b166a8e4a60fb374f69621595aea27fde34a1a0ae703 WatchSource:0}: Error finding container 8e9f126308e046f45340b166a8e4a60fb374f69621595aea27fde34a1a0ae703: Status 404 returned error can't find the container with id 8e9f126308e046f45340b166a8e4a60fb374f69621595aea27fde34a1a0ae703
Jul 02 13:46:41.499264 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.499229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:46:41.501990 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.501963    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aedca59b-0e89-4e5b-9092-3628bfc55297-metrics-tls\") pod \"dns-default-qkjj8\" (UID: \"aedca59b-0e89-4e5b-9092-3628bfc55297\") " pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:46:41.568194 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.568115    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t9b9z\""
Jul 02 13:46:41.576319 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.576282    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:46:41.738104 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.738042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qkjj8"]
Jul 02 13:46:41.742209 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:41.742183    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedca59b_0e89_4e5b_9092_3628bfc55297.slice/crio-cf6dc9ad905a1250be656bf00f8c1419c61c3dd471ff97a826ffd83aacaae7f6 WatchSource:0}: Error finding container cf6dc9ad905a1250be656bf00f8c1419c61c3dd471ff97a826ffd83aacaae7f6: Status 404 returned error can't find the container with id cf6dc9ad905a1250be656bf00f8c1419c61c3dd471ff97a826ffd83aacaae7f6
Jul 02 13:46:41.916677 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.916592    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qkjj8" event={"ID":"aedca59b-0e89-4e5b-9092-3628bfc55297","Type":"ContainerStarted","Data":"cf6dc9ad905a1250be656bf00f8c1419c61c3dd471ff97a826ffd83aacaae7f6"}
Jul 02 13:46:41.918464 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:41.918411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk" event={"ID":"0b6446cb-36a3-44d4-a51e-21547a9186c0","Type":"ContainerStarted","Data":"8e9f126308e046f45340b166a8e4a60fb374f69621595aea27fde34a1a0ae703"}
Jul 02 13:46:44.800171 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:44.800139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zn2d9"
Jul 02 13:46:45.935207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.935166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dc7dbb84-grc9m" event={"ID":"018d4c9d-e729-41f4-8d8d-3395172787bd","Type":"ContainerStarted","Data":"30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4"}
Jul 02 13:46:45.937214 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.937181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qkjj8" event={"ID":"aedca59b-0e89-4e5b-9092-3628bfc55297","Type":"ContainerStarted","Data":"1f30de492226163bd83edc5fa144c7f6562d164b3edaa13043fad163866fc365"}
Jul 02 13:46:45.937355 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.937256    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:46:45.937355 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.937281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qkjj8" event={"ID":"aedca59b-0e89-4e5b-9092-3628bfc55297","Type":"ContainerStarted","Data":"d0b645a21878646e8b32fd9dd6a36f365331b9b2d51180e5f1d35456f9a4f184"}
Jul 02 13:46:45.939073 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.939044    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk" event={"ID":"0b6446cb-36a3-44d4-a51e-21547a9186c0","Type":"ContainerStarted","Data":"ff0c3c252b4e5dc283d239fc97adae81e70ca0694f17a9725088fc41fbaca212"}
Jul 02 13:46:45.939187 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.939103    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"
Jul 02 13:46:45.944431 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.944400    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk"
Jul 02 13:46:45.955900 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.955864    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74dc7dbb84-grc9m" podStartSLOduration=1.947413349 podStartE2EDuration="6.955850668s" totalImagesPullingTime="5.008437319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:45.9541711 +0000 UTC m=+105.795808628" watchObservedRunningTime="2026-07-02 13:46:45.955850668 +0000 UTC m=+105.797488181"
Jul 02 13:46:45.976467 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.976431    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qkjj8" podStartSLOduration=65.653569839 podStartE2EDuration="1m8.97641549s" totalImagesPullingTime="3.322845651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:45.974653608 +0000 UTC m=+105.816291125" watchObservedRunningTime="2026-07-02 13:46:45.97641549 +0000 UTC m=+105.818053007"
Jul 02 13:46:45.993038 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:45.993003    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-v48zk" podStartSLOduration=2.3913552989999998 podStartE2EDuration="5.992992338s" totalImagesPullingTime="3.601637039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:45.992520972 +0000 UTC m=+105.834158498" watchObservedRunningTime="2026-07-02 13:46:45.992992338 +0000 UTC m=+105.834629830"
Jul 02 13:46:46.528456 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:46.528421    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:46:49.897796 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:49.897749    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:49.898261 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:49.897824    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:49.903154 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:49.903130    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:49.962803 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:49.962771    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:46:52.322541 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.322095    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"]
Jul 02 13:46:52.325331 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.325307    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.329677 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.329657    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:46:52.330760 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.330740    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xnkb8\""
Jul 02 13:46:52.330975 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.330953    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 02 13:46:52.331658 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.331460    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 13:46:52.376604 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.376496    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"]
Jul 02 13:46:52.383866 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.383664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3267367b-c780-49f0-8489-40fdf391a16b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.383866 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.383716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3267367b-c780-49f0-8489-40fdf391a16b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.383866 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.383752    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3267367b-c780-49f0-8489-40fdf391a16b-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.383866 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.383776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw64j\" (UniqueName: \"kubernetes.io/projected/3267367b-c780-49f0-8489-40fdf391a16b-kube-api-access-lw64j\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.416048 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.414464    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-mnjrf"]
Jul 02 13:46:52.417013 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.416992    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dhxp7"]
Jul 02 13:46:52.417150 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.417130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.419687 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.419667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.433121 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:52.432934    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-136-2.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret"
Jul 02 13:46:52.433121 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:52.433058    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-136-2.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret"
Jul 02 13:46:52.433287 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.433187    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-46zb5\""
Jul 02 13:46:52.434210 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.433430    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:46:52.434210 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.433655    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 02 13:46:52.434210 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:52.433855    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-136-2.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap"
Jul 02 13:46:52.434210 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:52.433919    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-6qtjc\" is forbidden: User \"system:node:ip-10-0-136-2.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-2.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6qtjc\"" type="*v1.Secret"
Jul 02 13:46:52.439440 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.439225    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 02 13:46:52.472438 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.472411    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-mnjrf"]
Jul 02 13:46:52.484860 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.484831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e460137-c685-4475-8d76-a750072a61be-node-exporter-accelerators-collector-config\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.484998 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.484881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sbwk\" (UniqueName: \"kubernetes.io/projected/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-api-access-2sbwk\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.484998 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.484918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3267367b-c780-49f0-8489-40fdf391a16b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.484998 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.484954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3267367b-c780-49f0-8489-40fdf391a16b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.484998 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.484982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e460137-c685-4475-8d76-a750072a61be-root\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.485207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e460137-c685-4475-8d76-a750072a61be-node-exporter-textfile\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.485207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e460137-c685-4475-8d76-a750072a61be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.485207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3267367b-c780-49f0-8489-40fdf391a16b-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.485207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw64j\" (UniqueName: \"kubernetes.io/projected/3267367b-c780-49f0-8489-40fdf391a16b-kube-api-access-lw64j\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.485207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.485207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.485207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e460137-c685-4475-8d76-a750072a61be-metrics-client-ca\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.485207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e460137-c685-4475-8d76-a750072a61be-sys\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.485990 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd69c4ab-d2a1-416d-af30-b0da27d96123-metrics-client-ca\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.485990 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fd69c4ab-d2a1-416d-af30-b0da27d96123-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.485990 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.485990 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.485355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8d9q\" (UniqueName: \"kubernetes.io/projected/0e460137-c685-4475-8d76-a750072a61be-kube-api-access-z8d9q\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.486821 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.486262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e460137-c685-4475-8d76-a750072a61be-node-exporter-wtmp\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.486821 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.486304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e460137-c685-4475-8d76-a750072a61be-node-exporter-tls\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.486821 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:52.486475    2566 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 02 13:46:52.486821 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:52.486670    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3267367b-c780-49f0-8489-40fdf391a16b-openshift-state-metrics-tls podName:3267367b-c780-49f0-8489-40fdf391a16b nodeName:}" failed. No retries permitted until 2026-07-02 13:46:52.9866485 +0000 UTC m=+112.828286005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3267367b-c780-49f0-8489-40fdf391a16b-openshift-state-metrics-tls") pod "openshift-state-metrics-57f99c8b49-sfs24" (UID: "3267367b-c780-49f0-8489-40fdf391a16b") : secret "openshift-state-metrics-tls" not found
Jul 02 13:46:52.487170 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.487114    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3267367b-c780-49f0-8489-40fdf391a16b-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.489828 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.489803    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3267367b-c780-49f0-8489-40fdf391a16b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.519026 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.518996    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw64j\" (UniqueName: \"kubernetes.io/projected/3267367b-c780-49f0-8489-40fdf391a16b-kube-api-access-lw64j\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.588100 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.587995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e460137-c685-4475-8d76-a750072a61be-root\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588100 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e460137-c685-4475-8d76-a750072a61be-node-exporter-textfile\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588100 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e460137-c685-4475-8d76-a750072a61be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588100 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.588398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588123    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e460137-c685-4475-8d76-a750072a61be-root\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.588398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e460137-c685-4475-8d76-a750072a61be-metrics-client-ca\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e460137-c685-4475-8d76-a750072a61be-sys\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd69c4ab-d2a1-416d-af30-b0da27d96123-metrics-client-ca\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.588398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fd69c4ab-d2a1-416d-af30-b0da27d96123-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.588398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.588398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8d9q\" (UniqueName: \"kubernetes.io/projected/0e460137-c685-4475-8d76-a750072a61be-kube-api-access-z8d9q\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588398 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e460137-c685-4475-8d76-a750072a61be-node-exporter-wtmp\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588891 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e460137-c685-4475-8d76-a750072a61be-node-exporter-tls\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588891 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e460137-c685-4475-8d76-a750072a61be-node-exporter-accelerators-collector-config\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.588891 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.588501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sbwk\" (UniqueName: \"kubernetes.io/projected/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-api-access-2sbwk\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.589059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.589028    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e460137-c685-4475-8d76-a750072a61be-node-exporter-wtmp\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.589166 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.589150    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e460137-c685-4475-8d76-a750072a61be-metrics-client-ca\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.589239 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.589219    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e460137-c685-4475-8d76-a750072a61be-sys\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.589528 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.589417    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fd69c4ab-d2a1-416d-af30-b0da27d96123-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.589528 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.589447    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd69c4ab-d2a1-416d-af30-b0da27d96123-metrics-client-ca\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.589685 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:52.589651    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 02 13:46:52.589743 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.589686    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e460137-c685-4475-8d76-a750072a61be-node-exporter-textfile\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.589743 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:52.589734    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-tls podName:fd69c4ab-d2a1-416d-af30-b0da27d96123 nodeName:}" failed. No retries permitted until 2026-07-02 13:46:53.089715482 +0000 UTC m=+112.931352993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-mnjrf" (UID: "fd69c4ab-d2a1-416d-af30-b0da27d96123") : secret "kube-state-metrics-tls" not found
Jul 02 13:46:52.589905 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.589880    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.591138 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.591109    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.607328 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.607305    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sbwk\" (UniqueName: \"kubernetes.io/projected/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-api-access-2sbwk\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:52.617819 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.617781    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8d9q\" (UniqueName: \"kubernetes.io/projected/0e460137-c685-4475-8d76-a750072a61be-kube-api-access-z8d9q\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:52.993320 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.993282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3267367b-c780-49f0-8489-40fdf391a16b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:52.996168 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:52.996141    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3267367b-c780-49f0-8489-40fdf391a16b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-sfs24\" (UID: \"3267367b-c780-49f0-8489-40fdf391a16b\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:53.093850 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.093810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:53.096628 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.096598    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd69c4ab-d2a1-416d-af30-b0da27d96123-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-mnjrf\" (UID: \"fd69c4ab-d2a1-416d-af30-b0da27d96123\") " pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:53.238402 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.238363    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"
Jul 02 13:46:53.335345 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.335259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf"
Jul 02 13:46:53.399496 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.399449    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 13:46:53.411167 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.411136    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e460137-c685-4475-8d76-a750072a61be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:53.563235 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.563201    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:46:53.571006 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.570979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.576982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.576915    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 02 13:46:53.577379 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.577251    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 02 13:46:53.577379 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.577317    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 02 13:46:53.577379 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.577355    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 02 13:46:53.577597 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.577384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 02 13:46:53.577666 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.577650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 02 13:46:53.577702 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.577666    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 02 13:46:53.577738 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.577698    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 02 13:46:53.577738 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.577718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 02 13:46:53.577808 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.577658    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6qjhm\""
Jul 02 13:46:53.589260 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:53.589209    2566 configmap.go:193] Couldn't get configMap openshift-monitoring/node-exporter-accelerators-collector-config: failed to sync configmap cache: timed out waiting for the condition
Jul 02 13:46:53.589260 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:53.589220    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: failed to sync secret cache: timed out waiting for the condition
Jul 02 13:46:53.589423 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:53.589275    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e460137-c685-4475-8d76-a750072a61be-node-exporter-tls podName:0e460137-c685-4475-8d76-a750072a61be nodeName:}" failed. No retries permitted until 2026-07-02 13:46:54.089260308 +0000 UTC m=+113.930897803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0e460137-c685-4475-8d76-a750072a61be-node-exporter-tls") pod "node-exporter-dhxp7" (UID: "0e460137-c685-4475-8d76-a750072a61be") : failed to sync secret cache: timed out waiting for the condition
Jul 02 13:46:53.589423 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:46:53.589300    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e460137-c685-4475-8d76-a750072a61be-node-exporter-accelerators-collector-config podName:0e460137-c685-4475-8d76-a750072a61be nodeName:}" failed. No retries permitted until 2026-07-02 13:46:54.089289991 +0000 UTC m=+113.930927485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-accelerators-collector-config" (UniqueName: "kubernetes.io/configmap/0e460137-c685-4475-8d76-a750072a61be-node-exporter-accelerators-collector-config") pod "node-exporter-dhxp7" (UID: "0e460137-c685-4475-8d76-a750072a61be") : failed to sync configmap cache: timed out waiting for the condition
Jul 02 13:46:53.598430 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.598394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcc60566-f8a4-4de6-aee3-a433e5090838-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.598573 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.598435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.598573 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.598462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.598573 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.598510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.598756 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.598723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-web-config\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.598863 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.598841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.599032 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.598934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcc60566-f8a4-4de6-aee3-a433e5090838-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.599032 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.598967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc60566-f8a4-4de6-aee3-a433e5090838-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.599032 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.598998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcc60566-f8a4-4de6-aee3-a433e5090838-config-out\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.599032 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.599025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnl5\" (UniqueName: \"kubernetes.io/projected/fcc60566-f8a4-4de6-aee3-a433e5090838-kube-api-access-9nnl5\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.599371 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.599109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-config-volume\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.599371 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.599133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcc60566-f8a4-4de6-aee3-a433e5090838-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.599504 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.599474    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:46:53.599683 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.599658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.683763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.683734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6qtjc\""
Jul 02 13:46:53.700227 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700399 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcc60566-f8a4-4de6-aee3-a433e5090838-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700399 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700399 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700399 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700399 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-web-config\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700399 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700720 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcc60566-f8a4-4de6-aee3-a433e5090838-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700720 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc60566-f8a4-4de6-aee3-a433e5090838-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700720 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcc60566-f8a4-4de6-aee3-a433e5090838-config-out\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700720 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnl5\" (UniqueName: \"kubernetes.io/projected/fcc60566-f8a4-4de6-aee3-a433e5090838-kube-api-access-9nnl5\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700720 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-config-volume\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.700720 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.700623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcc60566-f8a4-4de6-aee3-a433e5090838-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.701421 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.701044    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcc60566-f8a4-4de6-aee3-a433e5090838-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.701955 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.701928    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc60566-f8a4-4de6-aee3-a433e5090838-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.704104 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.704077    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcc60566-f8a4-4de6-aee3-a433e5090838-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.704932 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.704903    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.705351 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.705313    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-config-volume\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.705440 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.705392    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.705520 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.705435    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcc60566-f8a4-4de6-aee3-a433e5090838-config-out\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.705520 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.705466    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcc60566-f8a4-4de6-aee3-a433e5090838-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.705687 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.705660    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.705788 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.705752    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.705848 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.705832    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.706871 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.706848    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcc60566-f8a4-4de6-aee3-a433e5090838-web-config\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.721181 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.721107    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnl5\" (UniqueName: \"kubernetes.io/projected/fcc60566-f8a4-4de6-aee3-a433e5090838-kube-api-access-9nnl5\") pod \"alertmanager-main-0\" (UID: \"fcc60566-f8a4-4de6-aee3-a433e5090838\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:53.797538 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.797509    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 13:46:53.806205 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.806181    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 13:46:53.883591 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:53.883479    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:46:54.104253 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:54.104218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e460137-c685-4475-8d76-a750072a61be-node-exporter-tls\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:54.104430 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:54.104271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e460137-c685-4475-8d76-a750072a61be-node-exporter-accelerators-collector-config\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:54.104975 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:54.104921    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e460137-c685-4475-8d76-a750072a61be-node-exporter-accelerators-collector-config\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:54.106960 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:54.106937    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e460137-c685-4475-8d76-a750072a61be-node-exporter-tls\") pod \"node-exporter-dhxp7\" (UID: \"0e460137-c685-4475-8d76-a750072a61be\") " pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:54.241992 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:54.241957    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dhxp7"
Jul 02 13:46:54.781516 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:54.781471    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74dc7dbb84-grc9m"]
Jul 02 13:46:55.312084 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:55.312038    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e460137_c685_4475_8d76_a750072a61be.slice/crio-9c49864033e589903a18b62d9feb04dcd46371da5ff689ae3aa9a8923996ae37 WatchSource:0}: Error finding container 9c49864033e589903a18b62d9feb04dcd46371da5ff689ae3aa9a8923996ae37: Status 404 returned error can't find the container with id 9c49864033e589903a18b62d9feb04dcd46371da5ff689ae3aa9a8923996ae37
Jul 02 13:46:55.477338 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.476008    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"]
Jul 02 13:46:55.481552 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.481451    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.484354 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.484332    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 02 13:46:55.484746 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.484727    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 02 13:46:55.484829 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.484729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 02 13:46:55.484921 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.484897    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-238qt44mkhucm\""
Jul 02 13:46:55.485058 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.485041    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dts9n\""
Jul 02 13:46:55.485297 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.485281    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 02 13:46:55.485680 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.485663    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 02 13:46:55.501475 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.501454    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24"]
Jul 02 13:46:55.506607 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:55.506581    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3267367b_c780_49f0_8489_40fdf391a16b.slice/crio-f1956188a0e8c765b6e39dc7a13820e771d6428b1b366de2846031609e1218c7 WatchSource:0}: Error finding container f1956188a0e8c765b6e39dc7a13820e771d6428b1b366de2846031609e1218c7: Status 404 returned error can't find the container with id f1956188a0e8c765b6e39dc7a13820e771d6428b1b366de2846031609e1218c7
Jul 02 13:46:55.517236 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.513665    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"]
Jul 02 13:46:55.517469 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.517441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-metrics-client-ca\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.517585 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.517527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-tls\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.517642 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.517581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.517642 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.517613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.517751 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.517667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.517751 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.517711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.517751 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.517744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49mz8\" (UniqueName: \"kubernetes.io/projected/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-kube-api-access-49mz8\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.517885 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.517776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-grpc-tls\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.619189 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.619162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.619317 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.619240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.619317 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.619299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.619446 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.619338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49mz8\" (UniqueName: \"kubernetes.io/projected/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-kube-api-access-49mz8\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.619446 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.619368    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-grpc-tls\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.619446 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.619395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-metrics-client-ca\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.619446 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.619425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-tls\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.619649 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.619461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.620591 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.620536    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-metrics-client-ca\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.622919 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.622872    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.623411 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.623369    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.623535 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.623513    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-grpc-tls\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.623597 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.623560    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-tls\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.623959 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.623939    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.625267 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.625244    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.630626 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.630605    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49mz8\" (UniqueName: \"kubernetes.io/projected/3cc8d4be-b925-4fb5-a416-063d4ba8c9a6-kube-api-access-49mz8\") pod \"thanos-querier-5cb4f5765f-rlkt6\" (UID: \"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6\") " pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.709263 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.709129    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-mnjrf"]
Jul 02 13:46:55.711954 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:55.711739    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd69c4ab_d2a1_416d_af30_b0da27d96123.slice/crio-389b5c6a0100c00596fe1e9132a96dd3b2edcaa378fde45f5ced172f6e5d693e WatchSource:0}: Error finding container 389b5c6a0100c00596fe1e9132a96dd3b2edcaa378fde45f5ced172f6e5d693e: Status 404 returned error can't find the container with id 389b5c6a0100c00596fe1e9132a96dd3b2edcaa378fde45f5ced172f6e5d693e
Jul 02 13:46:55.715925 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.715893    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:46:55.722162 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:55.722135    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc60566_f8a4_4de6_aee3_a433e5090838.slice/crio-f20937d8963b0de08bd1e5432816ac9761e119dd049bdeace6d91932e969a9fc WatchSource:0}: Error finding container f20937d8963b0de08bd1e5432816ac9761e119dd049bdeace6d91932e969a9fc: Status 404 returned error can't find the container with id f20937d8963b0de08bd1e5432816ac9761e119dd049bdeace6d91932e969a9fc
Jul 02 13:46:55.803710 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.803673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:46:55.944695 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.944661    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qkjj8"
Jul 02 13:46:55.985732 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.985635    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"]
Jul 02 13:46:55.989377 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.988252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-n7nq7" event={"ID":"0a0dfc85-0d34-4338-a559-553b52c97cd2","Type":"ContainerStarted","Data":"2ffe71082fa292931687394181624d93873ef5e4eb9bc78ecdd0e1d59ef522b6"}
Jul 02 13:46:55.989377 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.988765    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-n7nq7"
Jul 02 13:46:55.991232 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:55.991206    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc8d4be_b925_4fb5_a416_063d4ba8c9a6.slice/crio-b27fb09a82d085dd6a7cc2770fbb938475ef8c9d764ac2d2d02cc08b051035ac WatchSource:0}: Error finding container b27fb09a82d085dd6a7cc2770fbb938475ef8c9d764ac2d2d02cc08b051035ac: Status 404 returned error can't find the container with id b27fb09a82d085dd6a7cc2770fbb938475ef8c9d764ac2d2d02cc08b051035ac
Jul 02 13:46:55.991792 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.991766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf" event={"ID":"fd69c4ab-d2a1-416d-af30-b0da27d96123","Type":"ContainerStarted","Data":"389b5c6a0100c00596fe1e9132a96dd3b2edcaa378fde45f5ced172f6e5d693e"}
Jul 02 13:46:55.994561 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.994474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-n7nq7"
Jul 02 13:46:55.996362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.996173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24" event={"ID":"3267367b-c780-49f0-8489-40fdf391a16b","Type":"ContainerStarted","Data":"9c578047196dec7e514f43922751c3145cde14568176f57edd58f2c38ad4b098"}
Jul 02 13:46:55.996362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.996227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24" event={"ID":"3267367b-c780-49f0-8489-40fdf391a16b","Type":"ContainerStarted","Data":"f549787d43bf5080e00b9b3ae387158d155aff2913db15486aea22c51bbe6a39"}
Jul 02 13:46:55.996362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.996259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24" event={"ID":"3267367b-c780-49f0-8489-40fdf391a16b","Type":"ContainerStarted","Data":"f1956188a0e8c765b6e39dc7a13820e771d6428b1b366de2846031609e1218c7"}
Jul 02 13:46:55.998368 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:55.998293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcc60566-f8a4-4de6-aee3-a433e5090838","Type":"ContainerStarted","Data":"f20937d8963b0de08bd1e5432816ac9761e119dd049bdeace6d91932e969a9fc"}
Jul 02 13:46:56.000400 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:56.000282    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dhxp7" event={"ID":"0e460137-c685-4475-8d76-a750072a61be","Type":"ContainerStarted","Data":"9c49864033e589903a18b62d9feb04dcd46371da5ff689ae3aa9a8923996ae37"}
Jul 02 13:46:56.012019 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:56.010574    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-n7nq7" podStartSLOduration=1.7489088179999999 podStartE2EDuration="20.010555367s" totalImagesPullingTime="18.261646549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:46:56.009440738 +0000 UTC m=+115.851078252" watchObservedRunningTime="2026-07-02 13:46:56.010555367 +0000 UTC m=+115.852192884"
Jul 02 13:46:57.009995 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.009895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6" event={"ID":"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6","Type":"ContainerStarted","Data":"b27fb09a82d085dd6a7cc2770fbb938475ef8c9d764ac2d2d02cc08b051035ac"}
Jul 02 13:46:57.156086 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.154589    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f85b75f7-zmvfs"]
Jul 02 13:46:57.172160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.171331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f85b75f7-zmvfs"]
Jul 02 13:46:57.172160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.171442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.181977 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.181599    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 13:46:57.237366 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.237334    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88sn7\" (UniqueName: \"kubernetes.io/projected/31f2fee5-df15-4365-962c-2deaa440c857-kube-api-access-88sn7\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.237558 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.237372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-trusted-ca-bundle\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.237558 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.237456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-oauth-serving-cert\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.237558 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.237474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-serving-cert\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.237558 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.237518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-service-ca\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.237762 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.237569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-console-config\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.237762 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.237645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-oauth-config\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.338921 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.338845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-serving-cert\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.338921 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.338890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-service-ca\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.339113 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.338936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-console-config\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.339113 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.338959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-oauth-config\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.339113 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.339008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88sn7\" (UniqueName: \"kubernetes.io/projected/31f2fee5-df15-4365-962c-2deaa440c857-kube-api-access-88sn7\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.339113 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.339033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-trusted-ca-bundle\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.339113 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.339100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-oauth-serving-cert\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.341190 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.340473    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-service-ca\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.341350 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.341187    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-oauth-serving-cert\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.343503 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.341658    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-console-config\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.348505 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.343885    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-oauth-config\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.350318 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.350293    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88sn7\" (UniqueName: \"kubernetes.io/projected/31f2fee5-df15-4365-962c-2deaa440c857-kube-api-access-88sn7\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.353086 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.353026    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-trusted-ca-bundle\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.353651 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.353631    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-serving-cert\") pod \"console-67f85b75f7-zmvfs\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") " pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:57.486890 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:57.486811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:46:58.473100 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.470741    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:46:58.501538 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.500819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.504783 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.504758    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 13:46:58.505239 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.505222    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 13:46:58.505470 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.505450    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8q2jf\""
Jul 02 13:46:58.508174 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.505734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 13:46:58.508174 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.505932    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 13:46:58.508174 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.506130    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 13:46:58.508174 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.506316    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 13:46:58.508174 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.507740    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 13:46:58.508174 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.507937    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 13:46:58.508174 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.507956    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 13:46:58.508510 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.508358    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 13:46:58.508958 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.508658    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 13:46:58.508958 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.508666    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 13:46:58.511165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.510389    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 13:46:58.511165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.510978    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 13:46:58.511884 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.511640    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4k7ma4morr99h\""
Jul 02 13:46:58.512709 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.512090    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:46:58.519283 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.519259    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 13:46:58.556703 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.556678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7h2\" (UniqueName: \"kubernetes.io/projected/ed426926-6db7-489c-8836-020e215de3ef-kube-api-access-bs7h2\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.556930 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.556723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.556930 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.556755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.556930 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.556777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.556930 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.556861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-config\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.556930 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.556910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.556949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.556983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed426926-6db7-489c-8836-020e215de3ef-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed426926-6db7-489c-8836-020e215de3ef-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.557362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.558015 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.557388    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.657958 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.657924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.658160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.657974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.658160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.658011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.658160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.658036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.658160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.658080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.658160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.658123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.658160 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.658149    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.658676 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.658178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.658676 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.658200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.658676 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.658254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7h2\" (UniqueName: \"kubernetes.io/projected/ed426926-6db7-489c-8836-020e215de3ef-kube-api-access-bs7h2\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.659806 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659110    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.659806 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.659806 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.659806 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659671    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.659806 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659589    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.659806 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659817    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-config\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.659995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.660032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed426926-6db7-489c-8836-020e215de3ef-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.660081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed426926-6db7-489c-8836-020e215de3ef-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.660111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.660138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660582 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.660221    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.660582 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.660455    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.661616 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.661586    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.670599 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.670437    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed426926-6db7-489c-8836-020e215de3ef-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.674477 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.672950    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-config\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.674477 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.673608    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.674477 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.674060    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.674477 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.674301    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed426926-6db7-489c-8836-020e215de3ef-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.674477 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.674441    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.677167 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.677119    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.678976 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.678935    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.685315 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.685270    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed426926-6db7-489c-8836-020e215de3ef-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.685408 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.685359    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.686031 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.685837    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.686031 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.686005    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7h2\" (UniqueName: \"kubernetes.io/projected/ed426926-6db7-489c-8836-020e215de3ef-kube-api-access-bs7h2\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.688573 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.688515    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed426926-6db7-489c-8836-020e215de3ef-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed426926-6db7-489c-8836-020e215de3ef\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:58.819325 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:58.819253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:46:59.005694 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:46:59.005605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f85b75f7-zmvfs"]
Jul 02 13:46:59.797878 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:46:59.797839    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f2fee5_df15_4365_962c_2deaa440c857.slice/crio-f867555e75644b5537becdbee92d886a2be393524551609feaf7be8000041be4 WatchSource:0}: Error finding container f867555e75644b5537becdbee92d886a2be393524551609feaf7be8000041be4: Status 404 returned error can't find the container with id f867555e75644b5537becdbee92d886a2be393524551609feaf7be8000041be4
Jul 02 13:47:00.029953 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:00.029914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f85b75f7-zmvfs" event={"ID":"31f2fee5-df15-4365-962c-2deaa440c857","Type":"ContainerStarted","Data":"f867555e75644b5537becdbee92d886a2be393524551609feaf7be8000041be4"}
Jul 02 13:47:00.216545 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:00.216515    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:47:01.036151 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.036117    2566 generic.go:356] "Generic (PLEG): container finished" podID="fcc60566-f8a4-4de6-aee3-a433e5090838" containerID="4d370d40499b9bfa9d4fac726c0735136ce8f7be39c551347372021c7b694c59" exitCode=0
Jul 02 13:47:01.036744 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.036213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcc60566-f8a4-4de6-aee3-a433e5090838","Type":"ContainerDied","Data":"4d370d40499b9bfa9d4fac726c0735136ce8f7be39c551347372021c7b694c59"}
Jul 02 13:47:01.038281 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.038253    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e460137-c685-4475-8d76-a750072a61be" containerID="f61734d600caf17bef47b0a23d217ddfb7e79258c7302b52996cce476ba56b97" exitCode=0
Jul 02 13:47:01.038422 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.038396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dhxp7" event={"ID":"0e460137-c685-4475-8d76-a750072a61be","Type":"ContainerDied","Data":"f61734d600caf17bef47b0a23d217ddfb7e79258c7302b52996cce476ba56b97"}
Jul 02 13:47:01.040116 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.040092    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed426926-6db7-489c-8836-020e215de3ef" containerID="15251cf490af004328c3b4000706d8dd50c61b42b7ba58307bb81581f776a9d0" exitCode=0
Jul 02 13:47:01.040207 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.040184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed426926-6db7-489c-8836-020e215de3ef","Type":"ContainerDied","Data":"15251cf490af004328c3b4000706d8dd50c61b42b7ba58307bb81581f776a9d0"}
Jul 02 13:47:01.040267 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.040253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed426926-6db7-489c-8836-020e215de3ef","Type":"ContainerStarted","Data":"24c54384003e10f822397a6f6f03b61b07367fb926653212de93b9230600ac38"}
Jul 02 13:47:01.045574 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.045547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f85b75f7-zmvfs" event={"ID":"31f2fee5-df15-4365-962c-2deaa440c857","Type":"ContainerStarted","Data":"3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875"}
Jul 02 13:47:01.048117 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.048081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf" event={"ID":"fd69c4ab-d2a1-416d-af30-b0da27d96123","Type":"ContainerStarted","Data":"20be896e937436108b37129f626a81f353a7257b69f7fe53b918eeca91853250"}
Jul 02 13:47:01.048212 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.048140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf" event={"ID":"fd69c4ab-d2a1-416d-af30-b0da27d96123","Type":"ContainerStarted","Data":"04b132d308a25fdb513813e3037805b4f236a7a5cb242e9c20de3daa090a80a7"}
Jul 02 13:47:01.048212 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.048167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf" event={"ID":"fd69c4ab-d2a1-416d-af30-b0da27d96123","Type":"ContainerStarted","Data":"58bdfc786623eba7082db0eb7827fd596740a1c9ab781b523663b3252dadd274"}
Jul 02 13:47:01.050537 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.050516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6" event={"ID":"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6","Type":"ContainerStarted","Data":"f7d6144e675c21c81bae103f1a11ee12f9abfb0be77471aafd392dd5db0bc52f"}
Jul 02 13:47:01.050632 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.050570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6" event={"ID":"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6","Type":"ContainerStarted","Data":"f61461508d09548bc699559ea9d7ec239c29c0e985831b3ea57d7e3e99db158e"}
Jul 02 13:47:01.050632 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.050608    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6" event={"ID":"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6","Type":"ContainerStarted","Data":"8587aa55035c95b1f17109c715af1b45cae4cda8fe35ab975bd6eb778e48f3a6"}
Jul 02 13:47:01.052974 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.052948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24" event={"ID":"3267367b-c780-49f0-8489-40fdf391a16b","Type":"ContainerStarted","Data":"7e3e3f8c70b06eaac283b6d5720b7e455836b4b782c7c9c31d34de4ba25ce17e"}
Jul 02 13:47:01.118741 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.117914    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-sfs24" podStartSLOduration=4.9562371469999995 podStartE2EDuration="9.117893512s" totalImagesPullingTime="4.161656365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:47:01.089168032 +0000 UTC m=+120.930805554" watchObservedRunningTime="2026-07-02 13:47:01.117893512 +0000 UTC m=+120.959531028"
Jul 02 13:47:01.118741 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.118535    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-mnjrf" podStartSLOduration=4.830791503 podStartE2EDuration="9.118525002s" totalImagesPullingTime="4.287733499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:47:01.115627356 +0000 UTC m=+120.957264873" watchObservedRunningTime="2026-07-02 13:47:01.118525002 +0000 UTC m=+120.960162518"
Jul 02 13:47:01.141210 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.141157    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67f85b75f7-zmvfs" podStartSLOduration=4.141139251 podStartE2EDuration="4.141139251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:47:01.139068612 +0000 UTC m=+120.980706127" watchObservedRunningTime="2026-07-02 13:47:01.141139251 +0000 UTC m=+120.982776764"
Jul 02 13:47:01.545748 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:01.545699    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p" podUID="7d1807cd-b1d5-4fdc-af58-5212bb8ae739" containerName="registry" containerID="cri-o://fe1ecd97144e2f935e06ee12df371785e61a7eb6cf10483ad7079723be972787" gracePeriod=30
Jul 02 13:47:02.064184 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.064142    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dhxp7" event={"ID":"0e460137-c685-4475-8d76-a750072a61be","Type":"ContainerStarted","Data":"c57faaeba0c287a2dc49a1a19f40241f55e253a75b866923447678904f156e46"}
Jul 02 13:47:02.064928 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.064252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dhxp7" event={"ID":"0e460137-c685-4475-8d76-a750072a61be","Type":"ContainerStarted","Data":"72117007439b5e7e736205c9dd6e474e92fdd51cc0ca9cbef218f21a2f14617f"}
Jul 02 13:47:02.068977 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.068942    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d1807cd-b1d5-4fdc-af58-5212bb8ae739" containerID="fe1ecd97144e2f935e06ee12df371785e61a7eb6cf10483ad7079723be972787" exitCode=0
Jul 02 13:47:02.069785 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.069173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p" event={"ID":"7d1807cd-b1d5-4fdc-af58-5212bb8ae739","Type":"ContainerDied","Data":"fe1ecd97144e2f935e06ee12df371785e61a7eb6cf10483ad7079723be972787"}
Jul 02 13:47:02.086964 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.086909    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dhxp7" podStartSLOduration=5.668629307 podStartE2EDuration="10.086887508s" totalImagesPullingTime="4.206699451s" totalInitContainerRuntime="211.55875ms" isStatefulPod=true podCreationTimestamp="2026-07-02 13:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:47:02.085321603 +0000 UTC m=+121.926959118" watchObservedRunningTime="2026-07-02 13:47:02.086887508 +0000 UTC m=+121.928525023"
Jul 02 13:47:02.135942 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.135906    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:47:02.203508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.203106    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-image-registry-private-configuration\") pod \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") "
Jul 02 13:47:02.203508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.203181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") pod \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") "
Jul 02 13:47:02.203508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.203228    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rsj7\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-kube-api-access-6rsj7\") pod \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") "
Jul 02 13:47:02.203508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.203272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-installation-pull-secrets\") pod \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") "
Jul 02 13:47:02.203508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.203323    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-ca-trust-extracted\") pod \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") "
Jul 02 13:47:02.203508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.203357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-certificates\") pod \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") "
Jul 02 13:47:02.203508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.203411    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-bound-sa-token\") pod \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") "
Jul 02 13:47:02.203508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.203438    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-trusted-ca\") pod \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\" (UID: \"7d1807cd-b1d5-4fdc-af58-5212bb8ae739\") "
Jul 02 13:47:02.204053 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.203937    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7d1807cd-b1d5-4fdc-af58-5212bb8ae739" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:47:02.205059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.204890    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7d1807cd-b1d5-4fdc-af58-5212bb8ae739" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:47:02.208295 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.208069    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-trusted-ca\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:02.209170 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.208747    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-certificates\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:02.210124 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.210080    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7d1807cd-b1d5-4fdc-af58-5212bb8ae739" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:47:02.223767 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.219099    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7d1807cd-b1d5-4fdc-af58-5212bb8ae739" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:47:02.223767 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.221549    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7d1807cd-b1d5-4fdc-af58-5212bb8ae739" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:47:02.223767 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.221616    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7d1807cd-b1d5-4fdc-af58-5212bb8ae739" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:47:02.223767 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.221854    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-kube-api-access-6rsj7" (OuterVolumeSpecName: "kube-api-access-6rsj7") pod "7d1807cd-b1d5-4fdc-af58-5212bb8ae739" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739"). InnerVolumeSpecName "kube-api-access-6rsj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:47:02.223767 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.223500    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7d1807cd-b1d5-4fdc-af58-5212bb8ae739" (UID: "7d1807cd-b1d5-4fdc-af58-5212bb8ae739"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:47:02.310269 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.310231    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-registry-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:02.310269 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.310265    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rsj7\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-kube-api-access-6rsj7\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:02.310269 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.310280    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-installation-pull-secrets\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:02.310595 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.310290    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-ca-trust-extracted\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:02.310595 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.310302    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-bound-sa-token\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:02.310595 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:02.310317    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d1807cd-b1d5-4fdc-af58-5212bb8ae739-image-registry-private-configuration\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:03.078420 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:03.078377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p" event={"ID":"7d1807cd-b1d5-4fdc-af58-5212bb8ae739","Type":"ContainerDied","Data":"9d1bdad1f8e7bc241ebedaed526cf6a1e702cd81a3305fb1b31c5d0c047b6987"}
Jul 02 13:47:03.078420 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:03.078450    2566 scope.go:117] "RemoveContainer" containerID="fe1ecd97144e2f935e06ee12df371785e61a7eb6cf10483ad7079723be972787"
Jul 02 13:47:03.079253 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:03.078705    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fb55f8df-spq4p"
Jul 02 13:47:03.087545 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:03.087411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6" event={"ID":"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6","Type":"ContainerStarted","Data":"72e3c9ff924110b91cefea92f662483533f0cabd26cb8b5752cfc4d1552f4986"}
Jul 02 13:47:03.099288 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:03.099223    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84fb55f8df-spq4p"]
Jul 02 13:47:03.105180 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:03.105134    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84fb55f8df-spq4p"]
Jul 02 13:47:03.567921 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:03.567220    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f85b75f7-zmvfs"]
Jul 02 13:47:04.095990 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:04.095904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcc60566-f8a4-4de6-aee3-a433e5090838","Type":"ContainerStarted","Data":"95a07b5f17e59eaf5572d826b6dae1141e6f83e0ebb9662d2bb1dfbf1a9ce22f"}
Jul 02 13:47:04.100902 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:04.100873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6" event={"ID":"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6","Type":"ContainerStarted","Data":"a06a8bbe3e1a548720e81883c37b76e32ee870f199eb9825eaa78e84b8139846"}
Jul 02 13:47:04.101049 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:04.100941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6" event={"ID":"3cc8d4be-b925-4fb5-a416-063d4ba8c9a6","Type":"ContainerStarted","Data":"013fa2879f15dbe4198428d76993114e8178fab392fd85d0979ad6673d959616"}
Jul 02 13:47:04.101261 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:04.101236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:47:04.127161 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:04.127117    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6" podStartSLOduration=3.178041374 podStartE2EDuration="9.127101303s" totalImagesPullingTime="5.949059929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:47:04.125232697 +0000 UTC m=+123.966870212" watchObservedRunningTime="2026-07-02 13:47:04.127101303 +0000 UTC m=+123.968738816"
Jul 02 13:47:04.505317 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:04.505266    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1807cd-b1d5-4fdc-af58-5212bb8ae739" path="/var/lib/kubelet/pods/7d1807cd-b1d5-4fdc-af58-5212bb8ae739/volumes"
Jul 02 13:47:05.112577 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:05.112523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5cb4f5765f-rlkt6"
Jul 02 13:47:06.113565 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.113537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed426926-6db7-489c-8836-020e215de3ef","Type":"ContainerStarted","Data":"4c4b3299e3ca78019c408309b79dec02ee756232df44a50de86a82ad0bdbd1d2"}
Jul 02 13:47:06.113900 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.113637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed426926-6db7-489c-8836-020e215de3ef","Type":"ContainerStarted","Data":"d8cf75ec6d65de7574b5acbb396e81f4dfb1048d043443ec44eeebe28b55150f"}
Jul 02 13:47:06.113900 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.113673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed426926-6db7-489c-8836-020e215de3ef","Type":"ContainerStarted","Data":"3e409dfb05f18e4151829be5c9b243717406a63ffa890f2a660aa5aad208b547"}
Jul 02 13:47:06.113900 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.113705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed426926-6db7-489c-8836-020e215de3ef","Type":"ContainerStarted","Data":"694eb2b70572659591bcb80db78f7684a95f082f4182e2754c3b67ae4eb9eb6a"}
Jul 02 13:47:06.113900 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.113730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed426926-6db7-489c-8836-020e215de3ef","Type":"ContainerStarted","Data":"9a57e4a8c38f229bf33a6c5b088a6c31400bc35eb32ddffa81a88fdf9f15a36e"}
Jul 02 13:47:06.116537 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.116511    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcc60566-f8a4-4de6-aee3-a433e5090838","Type":"ContainerStarted","Data":"4e7b480930ddb6a76986e702ce07aab3aa208107b6f340585052ddd2c879e366"}
Jul 02 13:47:06.116662 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.116568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcc60566-f8a4-4de6-aee3-a433e5090838","Type":"ContainerStarted","Data":"3b72b0887be283093645d53ba805dc15b39e246acfb20ccd56692b7adfda5d88"}
Jul 02 13:47:06.116662 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.116593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcc60566-f8a4-4de6-aee3-a433e5090838","Type":"ContainerStarted","Data":"a0d81db21c48a7915ec58732e49e185cf97e862abf742caf8139f7b0a34fd649"}
Jul 02 13:47:06.116662 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.116619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcc60566-f8a4-4de6-aee3-a433e5090838","Type":"ContainerStarted","Data":"332af5204a154e6ff810aed68a3dbe3d3fe496590a102a8dbb31f45f4a0a5931"}
Jul 02 13:47:06.116662 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.116642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcc60566-f8a4-4de6-aee3-a433e5090838","Type":"ContainerStarted","Data":"fcf80fce870891a83e455603ddc89b9c2df86c9fb422c7e507dfc5caed4e1fe6"}
Jul 02 13:47:06.143474 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:06.143415    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.092096401 podStartE2EDuration="13.14339497s" totalImagesPullingTime="7.017339377s" totalInitContainerRuntime="33.959192ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:47:06.142622891 +0000 UTC m=+125.984260415" watchObservedRunningTime="2026-07-02 13:47:06.14339497 +0000 UTC m=+125.985032486"
Jul 02 13:47:07.124067 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:07.124035    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed426926-6db7-489c-8836-020e215de3ef","Type":"ContainerStarted","Data":"51996a3a605f15909e1f56e9589cf3b1c8b3938a7173a64458ed7cfb5955b95f"}
Jul 02 13:47:07.168410 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:07.168355    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.739933562 podStartE2EDuration="9.168337895s" totalImagesPullingTime="4.4116143s" totalInitContainerRuntime="16.790033ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:47:07.167938525 +0000 UTC m=+127.009576040" watchObservedRunningTime="2026-07-02 13:47:07.168337895 +0000 UTC m=+127.009975408"
Jul 02 13:47:07.487689 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:07.487661    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:47:08.820099 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:08.820063    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:47:12.101416 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:12.101376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:47:12.104073 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:12.104042    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66cf39f9-4496-49ad-aabc-e9453dab385a-metrics-certs\") pod \"network-metrics-daemon-dm4g2\" (UID: \"66cf39f9-4496-49ad-aabc-e9453dab385a\") " pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:47:12.325695 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:12.325669    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9jh8m\""
Jul 02 13:47:12.333069 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:12.333045    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm4g2"
Jul 02 13:47:12.493145 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:12.493123    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dm4g2"]
Jul 02 13:47:12.495572 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:47:12.495547    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cf39f9_4496_49ad_aabc_e9453dab385a.slice/crio-db1ee5a29f6779684aecf9da6e14325747c6632b4d66e4ac2edd59fc8ee11828 WatchSource:0}: Error finding container db1ee5a29f6779684aecf9da6e14325747c6632b4d66e4ac2edd59fc8ee11828: Status 404 returned error can't find the container with id db1ee5a29f6779684aecf9da6e14325747c6632b4d66e4ac2edd59fc8ee11828
Jul 02 13:47:13.152100 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:13.152058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dm4g2" event={"ID":"66cf39f9-4496-49ad-aabc-e9453dab385a","Type":"ContainerStarted","Data":"db1ee5a29f6779684aecf9da6e14325747c6632b4d66e4ac2edd59fc8ee11828"}
Jul 02 13:47:14.158731 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:14.158704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dm4g2" event={"ID":"66cf39f9-4496-49ad-aabc-e9453dab385a","Type":"ContainerStarted","Data":"778e668e461dee09170bd8bae8c34129587002ba92299255c5b42f47e8408fa0"}
Jul 02 13:47:15.167852 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:15.167810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dm4g2" event={"ID":"66cf39f9-4496-49ad-aabc-e9453dab385a","Type":"ContainerStarted","Data":"86b0de240cc555d7b9847dfc7ed730050510b6f33c3462d04404028a751d814a"}
Jul 02 13:47:15.194105 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:15.194050    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dm4g2" podStartSLOduration=131.736062313 podStartE2EDuration="2m13.194033956s" totalImagesPullingTime="1.457971643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:47:15.191736605 +0000 UTC m=+135.033374123" watchObservedRunningTime="2026-07-02 13:47:15.194033956 +0000 UTC m=+135.035671470"
Jul 02 13:47:19.805029 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:19.804958    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-74dc7dbb84-grc9m" podUID="018d4c9d-e729-41f4-8d8d-3395172787bd" containerName="console" containerID="cri-o://30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4" gracePeriod=15
Jul 02 13:47:19.958896 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:19.958867    2566 patch_prober.go:28] interesting pod/console-74dc7dbb84-grc9m container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" start-of-body=
Jul 02 13:47:19.959021 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:19.958917    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-74dc7dbb84-grc9m" podUID="018d4c9d-e729-41f4-8d8d-3395172787bd" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused"
Jul 02 13:47:20.067374 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.067321    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:47:20.166416 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.166389    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-serving-cert\") pod \"018d4c9d-e729-41f4-8d8d-3395172787bd\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") "
Jul 02 13:47:20.166548 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.166432    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-oauth-config\") pod \"018d4c9d-e729-41f4-8d8d-3395172787bd\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") "
Jul 02 13:47:20.166548 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.166473    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-oauth-serving-cert\") pod \"018d4c9d-e729-41f4-8d8d-3395172787bd\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") "
Jul 02 13:47:20.166548 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.166534    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts5hw\" (UniqueName: \"kubernetes.io/projected/018d4c9d-e729-41f4-8d8d-3395172787bd-kube-api-access-ts5hw\") pod \"018d4c9d-e729-41f4-8d8d-3395172787bd\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") "
Jul 02 13:47:20.166713 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.166554    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-console-config\") pod \"018d4c9d-e729-41f4-8d8d-3395172787bd\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") "
Jul 02 13:47:20.166713 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.166574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-service-ca\") pod \"018d4c9d-e729-41f4-8d8d-3395172787bd\" (UID: \"018d4c9d-e729-41f4-8d8d-3395172787bd\") "
Jul 02 13:47:20.166962 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.166928    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "018d4c9d-e729-41f4-8d8d-3395172787bd" (UID: "018d4c9d-e729-41f4-8d8d-3395172787bd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:47:20.167066 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.166942    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-console-config" (OuterVolumeSpecName: "console-config") pod "018d4c9d-e729-41f4-8d8d-3395172787bd" (UID: "018d4c9d-e729-41f4-8d8d-3395172787bd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:47:20.167066 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.166985    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-service-ca" (OuterVolumeSpecName: "service-ca") pod "018d4c9d-e729-41f4-8d8d-3395172787bd" (UID: "018d4c9d-e729-41f4-8d8d-3395172787bd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:47:20.168766 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.168737    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "018d4c9d-e729-41f4-8d8d-3395172787bd" (UID: "018d4c9d-e729-41f4-8d8d-3395172787bd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:47:20.168867 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.168777    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "018d4c9d-e729-41f4-8d8d-3395172787bd" (UID: "018d4c9d-e729-41f4-8d8d-3395172787bd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:47:20.168867 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.168849    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018d4c9d-e729-41f4-8d8d-3395172787bd-kube-api-access-ts5hw" (OuterVolumeSpecName: "kube-api-access-ts5hw") pod "018d4c9d-e729-41f4-8d8d-3395172787bd" (UID: "018d4c9d-e729-41f4-8d8d-3395172787bd"). InnerVolumeSpecName "kube-api-access-ts5hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:47:20.197934 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.197908    2566 generic.go:356] "Generic (PLEG): container finished" podID="018d4c9d-e729-41f4-8d8d-3395172787bd" containerID="30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4" exitCode=0
Jul 02 13:47:20.198044 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.197950    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dc7dbb84-grc9m"
Jul 02 13:47:20.198044 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.197964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dc7dbb84-grc9m" event={"ID":"018d4c9d-e729-41f4-8d8d-3395172787bd","Type":"ContainerDied","Data":"30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4"}
Jul 02 13:47:20.198044 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.198016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dc7dbb84-grc9m" event={"ID":"018d4c9d-e729-41f4-8d8d-3395172787bd","Type":"ContainerDied","Data":"683f7567c771c2f3a73601f7a6f10301a1493afa74285e39b15d302470064576"}
Jul 02 13:47:20.198200 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.198043    2566 scope.go:117] "RemoveContainer" containerID="30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4"
Jul 02 13:47:20.206845 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.206831    2566 scope.go:117] "RemoveContainer" containerID="30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4"
Jul 02 13:47:20.207082 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:47:20.207064    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4\": container with ID starting with 30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4 not found: ID does not exist" containerID="30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4"
Jul 02 13:47:20.207130 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.207089    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4"} err="failed to get container status \"30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4\": rpc error: code = NotFound desc = could not find container \"30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4\": container with ID starting with 30e6b00b29dce09e5e204904614374d60a5edb48b5221ae005d4052b4a3e56c4 not found: ID does not exist"
Jul 02 13:47:20.222873 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.222850    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74dc7dbb84-grc9m"]
Jul 02 13:47:20.226525 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.226505    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74dc7dbb84-grc9m"]
Jul 02 13:47:20.267293 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.267272    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-serving-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:20.267293 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.267291    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/018d4c9d-e729-41f4-8d8d-3395172787bd-console-oauth-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:20.267406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.267302    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-oauth-serving-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:20.267406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.267311    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5hw\" (UniqueName: \"kubernetes.io/projected/018d4c9d-e729-41f4-8d8d-3395172787bd-kube-api-access-ts5hw\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:20.267406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.267320    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-console-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:20.267406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.267328    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/018d4c9d-e729-41f4-8d8d-3395172787bd-service-ca\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:20.503734 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:20.503709    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="018d4c9d-e729-41f4-8d8d-3395172787bd" path="/var/lib/kubelet/pods/018d4c9d-e729-41f4-8d8d-3395172787bd/volumes"
Jul 02 13:47:25.230216 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:25.230186    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3d9299e-86c3-4cf4-9464-6a3cb72e0287" containerID="c3fc37209edde8c2269a75599594e3acb00daba3a54f1754ca65193540c5c4e9" exitCode=0
Jul 02 13:47:25.230579 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:25.230256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl" event={"ID":"e3d9299e-86c3-4cf4-9464-6a3cb72e0287","Type":"ContainerDied","Data":"c3fc37209edde8c2269a75599594e3acb00daba3a54f1754ca65193540c5c4e9"}
Jul 02 13:47:25.230625 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:25.230590    2566 scope.go:117] "RemoveContainer" containerID="c3fc37209edde8c2269a75599594e3acb00daba3a54f1754ca65193540c5c4e9"
Jul 02 13:47:26.236857 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:26.236819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-v7nvl" event={"ID":"e3d9299e-86c3-4cf4-9464-6a3cb72e0287","Type":"ContainerStarted","Data":"6bd595da896072e884ad0f9474362c4d9d58b0986e7285ff6386016f3a2e5e8c"}
Jul 02 13:47:28.593005 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:28.592962    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-67f85b75f7-zmvfs" podUID="31f2fee5-df15-4365-962c-2deaa440c857" containerName="console" containerID="cri-o://3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875" gracePeriod=15
Jul 02 13:47:28.835308 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:28.835282    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:47:29.032382 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.032356    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-serving-cert\") pod \"31f2fee5-df15-4365-962c-2deaa440c857\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") "
Jul 02 13:47:29.032532 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.032394    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-trusted-ca-bundle\") pod \"31f2fee5-df15-4365-962c-2deaa440c857\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") "
Jul 02 13:47:29.032532 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.032422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-oauth-config\") pod \"31f2fee5-df15-4365-962c-2deaa440c857\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") "
Jul 02 13:47:29.032626 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.032548    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88sn7\" (UniqueName: \"kubernetes.io/projected/31f2fee5-df15-4365-962c-2deaa440c857-kube-api-access-88sn7\") pod \"31f2fee5-df15-4365-962c-2deaa440c857\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") "
Jul 02 13:47:29.032626 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.032577    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-oauth-serving-cert\") pod \"31f2fee5-df15-4365-962c-2deaa440c857\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") "
Jul 02 13:47:29.032626 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.032598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-console-config\") pod \"31f2fee5-df15-4365-962c-2deaa440c857\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") "
Jul 02 13:47:29.032772 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.032632    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-service-ca\") pod \"31f2fee5-df15-4365-962c-2deaa440c857\" (UID: \"31f2fee5-df15-4365-962c-2deaa440c857\") "
Jul 02 13:47:29.032945 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.032909    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "31f2fee5-df15-4365-962c-2deaa440c857" (UID: "31f2fee5-df15-4365-962c-2deaa440c857"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:47:29.033066 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.033029    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "31f2fee5-df15-4365-962c-2deaa440c857" (UID: "31f2fee5-df15-4365-962c-2deaa440c857"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:47:29.033172 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.033042    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-console-config" (OuterVolumeSpecName: "console-config") pod "31f2fee5-df15-4365-962c-2deaa440c857" (UID: "31f2fee5-df15-4365-962c-2deaa440c857"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:47:29.033276 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.033217    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-service-ca" (OuterVolumeSpecName: "service-ca") pod "31f2fee5-df15-4365-962c-2deaa440c857" (UID: "31f2fee5-df15-4365-962c-2deaa440c857"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:47:29.034819 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.034792    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f2fee5-df15-4365-962c-2deaa440c857-kube-api-access-88sn7" (OuterVolumeSpecName: "kube-api-access-88sn7") pod "31f2fee5-df15-4365-962c-2deaa440c857" (UID: "31f2fee5-df15-4365-962c-2deaa440c857"). InnerVolumeSpecName "kube-api-access-88sn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:47:29.035097 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.035077    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "31f2fee5-df15-4365-962c-2deaa440c857" (UID: "31f2fee5-df15-4365-962c-2deaa440c857"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:47:29.035156 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.035094    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "31f2fee5-df15-4365-962c-2deaa440c857" (UID: "31f2fee5-df15-4365-962c-2deaa440c857"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:47:29.133313 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.133291    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-serving-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:29.133313 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.133312    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-trusted-ca-bundle\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:29.133443 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.133326    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31f2fee5-df15-4365-962c-2deaa440c857-console-oauth-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:29.133443 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.133339    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88sn7\" (UniqueName: \"kubernetes.io/projected/31f2fee5-df15-4365-962c-2deaa440c857-kube-api-access-88sn7\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:29.133443 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.133349    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-oauth-serving-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:29.133443 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.133357    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-console-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:29.133443 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.133366    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31f2fee5-df15-4365-962c-2deaa440c857-service-ca\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:47:29.252965 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.252939    2566 generic.go:356] "Generic (PLEG): container finished" podID="31f2fee5-df15-4365-962c-2deaa440c857" containerID="3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875" exitCode=0
Jul 02 13:47:29.253061 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.252986    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f85b75f7-zmvfs"
Jul 02 13:47:29.253061 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.253007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f85b75f7-zmvfs" event={"ID":"31f2fee5-df15-4365-962c-2deaa440c857","Type":"ContainerDied","Data":"3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875"}
Jul 02 13:47:29.253061 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.253051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f85b75f7-zmvfs" event={"ID":"31f2fee5-df15-4365-962c-2deaa440c857","Type":"ContainerDied","Data":"f867555e75644b5537becdbee92d886a2be393524551609feaf7be8000041be4"}
Jul 02 13:47:29.253177 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.253072    2566 scope.go:117] "RemoveContainer" containerID="3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875"
Jul 02 13:47:29.264173 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.264153    2566 scope.go:117] "RemoveContainer" containerID="3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875"
Jul 02 13:47:29.264459 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:47:29.264438    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875\": container with ID starting with 3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875 not found: ID does not exist" containerID="3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875"
Jul 02 13:47:29.264560 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.264473    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875"} err="failed to get container status \"3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875\": rpc error: code = NotFound desc = could not find container \"3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875\": container with ID starting with 3473fdc5915719527bc93b4d856b0b1dce7f0c25a2f02d4fa1ef85af5e44e875 not found: ID does not exist"
Jul 02 13:47:29.278914 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.278886    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f85b75f7-zmvfs"]
Jul 02 13:47:29.284247 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:29.284198    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f85b75f7-zmvfs"]
Jul 02 13:47:30.260609 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:30.260577    2566 generic.go:356] "Generic (PLEG): container finished" podID="ef9709a9-9047-412d-a162-071bb4647e73" containerID="74c5e84ed41463a92b2f5dd0a304e77d453b7e5cd462dc677c01795f58bcd0fd" exitCode=0
Jul 02 13:47:30.260967 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:30.260656    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h" event={"ID":"ef9709a9-9047-412d-a162-071bb4647e73","Type":"ContainerDied","Data":"74c5e84ed41463a92b2f5dd0a304e77d453b7e5cd462dc677c01795f58bcd0fd"}
Jul 02 13:47:30.261017 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:30.260989    2566 scope.go:117] "RemoveContainer" containerID="74c5e84ed41463a92b2f5dd0a304e77d453b7e5cd462dc677c01795f58bcd0fd"
Jul 02 13:47:30.503769 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:30.503744    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f2fee5-df15-4365-962c-2deaa440c857" path="/var/lib/kubelet/pods/31f2fee5-df15-4365-962c-2deaa440c857/volumes"
Jul 02 13:47:31.266458 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:31.266428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-6wr2h" event={"ID":"ef9709a9-9047-412d-a162-071bb4647e73","Type":"ContainerStarted","Data":"22e474d820ea5c93bf4666a021c07c2b185ec55d166d0f6f8d86ba176a1e4ba3"}
Jul 02 13:47:58.819936 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:58.819828    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:47:58.841268 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:58.841238    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:47:59.423876 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:47:59.423847    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:50:01.135665 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:50:01.135631    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 13:50:01.142763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:50:01.142738    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 13:50:01.142998 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:50:01.142981    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 13:52:24.821568 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821493    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-kmlj8"]
Jul 02 13:52:24.821982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821809    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f2fee5-df15-4365-962c-2deaa440c857" containerName="console"
Jul 02 13:52:24.821982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821819    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f2fee5-df15-4365-962c-2deaa440c857" containerName="console"
Jul 02 13:52:24.821982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821835    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1807cd-b1d5-4fdc-af58-5212bb8ae739" containerName="registry"
Jul 02 13:52:24.821982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821842    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1807cd-b1d5-4fdc-af58-5212bb8ae739" containerName="registry"
Jul 02 13:52:24.821982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821858    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018d4c9d-e729-41f4-8d8d-3395172787bd" containerName="console"
Jul 02 13:52:24.821982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821863    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="018d4c9d-e729-41f4-8d8d-3395172787bd" containerName="console"
Jul 02 13:52:24.821982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821924    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="018d4c9d-e729-41f4-8d8d-3395172787bd" containerName="console"
Jul 02 13:52:24.821982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821932    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1807cd-b1d5-4fdc-af58-5212bb8ae739" containerName="registry"
Jul 02 13:52:24.821982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.821938    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f2fee5-df15-4365-962c-2deaa440c857" containerName="console"
Jul 02 13:52:24.823939 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.823922    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kmlj8"
Jul 02 13:52:24.826335 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.826317    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 13:52:24.827020 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.827003    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-m9vvx\""
Jul 02 13:52:24.827110 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.827018    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 13:52:24.827110 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.827054    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 13:52:24.833103 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.833081    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kmlj8"]
Jul 02 13:52:24.932853 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.932822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpc9\" (UniqueName: \"kubernetes.io/projected/b2612dae-1721-4e5d-bdbf-b419d77189ec-kube-api-access-btpc9\") pod \"s3-init-kmlj8\" (UID: \"b2612dae-1721-4e5d-bdbf-b419d77189ec\") " pod="kserve/s3-init-kmlj8"
Jul 02 13:52:24.932990 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:24.932876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2612dae-1721-4e5d-bdbf-b419d77189ec-model-cache\") pod \"s3-init-kmlj8\" (UID: \"b2612dae-1721-4e5d-bdbf-b419d77189ec\") " pod="kserve/s3-init-kmlj8"
Jul 02 13:52:25.033827 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:25.033800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2612dae-1721-4e5d-bdbf-b419d77189ec-model-cache\") pod \"s3-init-kmlj8\" (UID: \"b2612dae-1721-4e5d-bdbf-b419d77189ec\") " pod="kserve/s3-init-kmlj8"
Jul 02 13:52:25.033937 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:25.033851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpc9\" (UniqueName: \"kubernetes.io/projected/b2612dae-1721-4e5d-bdbf-b419d77189ec-kube-api-access-btpc9\") pod \"s3-init-kmlj8\" (UID: \"b2612dae-1721-4e5d-bdbf-b419d77189ec\") " pod="kserve/s3-init-kmlj8"
Jul 02 13:52:25.034179 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:25.034160    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2612dae-1721-4e5d-bdbf-b419d77189ec-model-cache\") pod \"s3-init-kmlj8\" (UID: \"b2612dae-1721-4e5d-bdbf-b419d77189ec\") " pod="kserve/s3-init-kmlj8"
Jul 02 13:52:25.042228 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:25.042202    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpc9\" (UniqueName: \"kubernetes.io/projected/b2612dae-1721-4e5d-bdbf-b419d77189ec-kube-api-access-btpc9\") pod \"s3-init-kmlj8\" (UID: \"b2612dae-1721-4e5d-bdbf-b419d77189ec\") " pod="kserve/s3-init-kmlj8"
Jul 02 13:52:25.139163 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:25.139110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kmlj8"
Jul 02 13:52:25.267092 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:25.267068    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kmlj8"]
Jul 02 13:52:25.269519 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:52:25.269494    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2612dae_1721_4e5d_bdbf_b419d77189ec.slice/crio-690d040cf4605d7dc36b32343c146cdcc0c0c03d2b4391025fe1ab927ff6a77a WatchSource:0}: Error finding container 690d040cf4605d7dc36b32343c146cdcc0c0c03d2b4391025fe1ab927ff6a77a: Status 404 returned error can't find the container with id 690d040cf4605d7dc36b32343c146cdcc0c0c03d2b4391025fe1ab927ff6a77a
Jul 02 13:52:25.271430 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:25.271413    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:52:25.550446 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:25.550416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kmlj8" event={"ID":"b2612dae-1721-4e5d-bdbf-b419d77189ec","Type":"ContainerStarted","Data":"690d040cf4605d7dc36b32343c146cdcc0c0c03d2b4391025fe1ab927ff6a77a"}
Jul 02 13:52:29.571426 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:29.571387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kmlj8" event={"ID":"b2612dae-1721-4e5d-bdbf-b419d77189ec","Type":"ContainerStarted","Data":"4b230ccaf13e62db121e08618ad65c6b81678dc845e40b68543debfcd0370131"}
Jul 02 13:52:30.576954 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:30.576922    2566 generic.go:356] "Generic (PLEG): container finished" podID="b2612dae-1721-4e5d-bdbf-b419d77189ec" containerID="4b230ccaf13e62db121e08618ad65c6b81678dc845e40b68543debfcd0370131" exitCode=0
Jul 02 13:52:30.577311 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:30.576972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kmlj8" event={"ID":"b2612dae-1721-4e5d-bdbf-b419d77189ec","Type":"ContainerDied","Data":"4b230ccaf13e62db121e08618ad65c6b81678dc845e40b68543debfcd0370131"}
Jul 02 13:52:35.604712 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:35.604636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kmlj8" event={"ID":"b2612dae-1721-4e5d-bdbf-b419d77189ec","Type":"ContainerStarted","Data":"15a8b89e886e53311eebc5bab2755bcad4c380b645fe085caef2d7edf7ebaec8"}
Jul 02 13:52:35.623549 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:52:35.623478    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-kmlj8" podStartSLOduration=2.2701968040000002 podStartE2EDuration="11.623464699s" totalImagesPullingTime="7.895259088s" totalInitContainerRuntime="1.458008807s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:52:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:52:35.62101139 +0000 UTC m=+455.462648903" watchObservedRunningTime="2026-07-02 13:52:35.623464699 +0000 UTC m=+455.465102212"
Jul 02 13:53:40.893528 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:40.893470    2566 generic.go:356] "Generic (PLEG): container finished" podID="b2612dae-1721-4e5d-bdbf-b419d77189ec" containerID="15a8b89e886e53311eebc5bab2755bcad4c380b645fe085caef2d7edf7ebaec8" exitCode=0
Jul 02 13:53:40.893953 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:40.893557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kmlj8" event={"ID":"b2612dae-1721-4e5d-bdbf-b419d77189ec","Type":"ContainerDied","Data":"15a8b89e886e53311eebc5bab2755bcad4c380b645fe085caef2d7edf7ebaec8"}
Jul 02 13:53:42.030489 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.030464    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kmlj8"
Jul 02 13:53:42.119622 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.119596    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btpc9\" (UniqueName: \"kubernetes.io/projected/b2612dae-1721-4e5d-bdbf-b419d77189ec-kube-api-access-btpc9\") pod \"b2612dae-1721-4e5d-bdbf-b419d77189ec\" (UID: \"b2612dae-1721-4e5d-bdbf-b419d77189ec\") "
Jul 02 13:53:42.119734 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.119700    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2612dae-1721-4e5d-bdbf-b419d77189ec-model-cache\") pod \"b2612dae-1721-4e5d-bdbf-b419d77189ec\" (UID: \"b2612dae-1721-4e5d-bdbf-b419d77189ec\") "
Jul 02 13:53:42.121636 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.121609    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2612dae-1721-4e5d-bdbf-b419d77189ec-kube-api-access-btpc9" (OuterVolumeSpecName: "kube-api-access-btpc9") pod "b2612dae-1721-4e5d-bdbf-b419d77189ec" (UID: "b2612dae-1721-4e5d-bdbf-b419d77189ec"). InnerVolumeSpecName "kube-api-access-btpc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:53:42.138292 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.138267    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2612dae-1721-4e5d-bdbf-b419d77189ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2612dae-1721-4e5d-bdbf-b419d77189ec" (UID: "b2612dae-1721-4e5d-bdbf-b419d77189ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:53:42.221075 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.221052    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2612dae-1721-4e5d-bdbf-b419d77189ec-model-cache\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:53:42.221075 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.221073    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpc9\" (UniqueName: \"kubernetes.io/projected/b2612dae-1721-4e5d-bdbf-b419d77189ec-kube-api-access-btpc9\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:53:42.903942 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.903873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kmlj8" event={"ID":"b2612dae-1721-4e5d-bdbf-b419d77189ec","Type":"ContainerDied","Data":"690d040cf4605d7dc36b32343c146cdcc0c0c03d2b4391025fe1ab927ff6a77a"}
Jul 02 13:53:42.903942 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.903910    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kmlj8"
Jul 02 13:53:42.904086 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:42.903915    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="690d040cf4605d7dc36b32343c146cdcc0c0c03d2b4391025fe1ab927ff6a77a"
Jul 02 13:53:54.993411 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:54.993315    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"]
Jul 02 13:53:54.993780 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:54.993677    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2612dae-1721-4e5d-bdbf-b419d77189ec" containerName="download-hf-model"
Jul 02 13:53:54.993780 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:54.993689    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2612dae-1721-4e5d-bdbf-b419d77189ec" containerName="download-hf-model"
Jul 02 13:53:54.993780 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:54.993713    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2612dae-1721-4e5d-bdbf-b419d77189ec" containerName="s3-init"
Jul 02 13:53:54.993780 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:54.993718    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2612dae-1721-4e5d-bdbf-b419d77189ec" containerName="s3-init"
Jul 02 13:53:54.993780 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:54.993771    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2612dae-1721-4e5d-bdbf-b419d77189ec" containerName="s3-init"
Jul 02 13:53:54.996945 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:54.996930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.000611 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.000592    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2cb97-predictor-serving-cert\""
Jul 02 13:53:55.000760 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.000744    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:53:55.000831 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.000804    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:53:55.001049 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.001019    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2cb97-kube-rbac-proxy-sar-config\""
Jul 02 13:53:55.001049 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.001050    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-bxgtw\""
Jul 02 13:53:55.002938 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.002921    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:53:55.003541 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.003516    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:53:55.023654 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.023626    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"]
Jul 02 13:53:55.120355 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.120329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.120549 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.120373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb28b844-62f9-408a-8113-4168015658c1-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.120549 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.120447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.120679 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.120568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szvmh\" (UniqueName: \"kubernetes.io/projected/cb28b844-62f9-408a-8113-4168015658c1-kube-api-access-szvmh\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.120679 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.120624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb28b844-62f9-408a-8113-4168015658c1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.221670 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.221643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.221872 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.221854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szvmh\" (UniqueName: \"kubernetes.io/projected/cb28b844-62f9-408a-8113-4168015658c1-kube-api-access-szvmh\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.222011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.221996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb28b844-62f9-408a-8113-4168015658c1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.222134 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.222120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.222245 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.222232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb28b844-62f9-408a-8113-4168015658c1-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.222797 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.222776    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb28b844-62f9-408a-8113-4168015658c1-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.223572 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.223550    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.224122 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:53:55.224102    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-2cb97-predictor-serving-cert" not found
Jul 02 13:53:55.224315 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:53:55.224304    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb28b844-62f9-408a-8113-4168015658c1-proxy-tls podName:cb28b844-62f9-408a-8113-4168015658c1 nodeName:}" failed. No retries permitted until 2026-07-02 13:53:55.724283943 +0000 UTC m=+535.565921435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cb28b844-62f9-408a-8113-4168015658c1-proxy-tls") pod "isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" (UID: "cb28b844-62f9-408a-8113-4168015658c1") : secret "isvc-raw-sklearn-batcher-2cb97-predictor-serving-cert" not found
Jul 02 13:53:55.225215 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.225193    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.236083 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.236064    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szvmh\" (UniqueName: \"kubernetes.io/projected/cb28b844-62f9-408a-8113-4168015658c1-kube-api-access-szvmh\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.725563 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.725533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb28b844-62f9-408a-8113-4168015658c1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.728074 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.728044    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb28b844-62f9-408a-8113-4168015658c1-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:55.909449 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:55.909418    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:53:56.044615 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:56.044592    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"]
Jul 02 13:53:56.046541 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:53:56.046516    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb28b844_62f9_408a_8113_4168015658c1.slice/crio-230762d6bdef92991d66875fb84aab4d40b3ca4411ce703661710f8540a391d7 WatchSource:0}: Error finding container 230762d6bdef92991d66875fb84aab4d40b3ca4411ce703661710f8540a391d7: Status 404 returned error can't find the container with id 230762d6bdef92991d66875fb84aab4d40b3ca4411ce703661710f8540a391d7
Jul 02 13:53:56.971103 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:56.971062    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerStarted","Data":"53678a5220cbf5427c66a01b94ca6b385037a8c735605b1567fc295b5d960935"}
Jul 02 13:53:56.971315 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:56.971131    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerStarted","Data":"230762d6bdef92991d66875fb84aab4d40b3ca4411ce703661710f8540a391d7"}
Jul 02 13:53:59.984968 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:59.984937    2566 generic.go:356] "Generic (PLEG): container finished" podID="cb28b844-62f9-408a-8113-4168015658c1" containerID="53678a5220cbf5427c66a01b94ca6b385037a8c735605b1567fc295b5d960935" exitCode=0
Jul 02 13:53:59.985415 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:53:59.985014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerDied","Data":"53678a5220cbf5427c66a01b94ca6b385037a8c735605b1567fc295b5d960935"}
Jul 02 13:54:13.060385 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:13.060355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerStarted","Data":"7f672f841be14faafdcc4427bc04c530075e81a56ac40139cea25bc15d4b9a91"}
Jul 02 13:54:16.080428 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:16.080394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerStarted","Data":"fc11010b8079ca5deed24e95f85b07fc0e8911c509a78c739df9748b2663be4a"}
Jul 02 13:54:18.091156 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:18.091083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerStarted","Data":"78ef739b566384e788353dd843a9b13fc460fef66c6f491fd78e3bf7bc8fdc11"}
Jul 02 13:54:18.091574 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:18.091314    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:54:18.091574 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:18.091467    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:54:18.092406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:18.092384    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:54:18.114508 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:18.114454    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podStartSLOduration=2.831160201 podStartE2EDuration="24.114442168s" totalImagesPullingTime="17.587668247s" totalInitContainerRuntime="3.69561372s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:53:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:54:18.112575195 +0000 UTC m=+557.954212708" watchObservedRunningTime="2026-07-02 13:54:18.114442168 +0000 UTC m=+557.956079682"
Jul 02 13:54:19.095815 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:19.095783    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:54:19.096205 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:19.095968    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:54:19.096915 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:19.096888    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:54:20.099886 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:20.099844    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:54:20.100301 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:20.100197    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:54:20.103602 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:20.103583    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:54:21.103528 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:21.103468    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:54:21.103897 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:21.103723    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:54:31.103888 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:31.103847    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:54:31.104376 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:31.104350    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:54:41.103950 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:41.103916    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:54:41.104504 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:41.104463    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:54:51.103444 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:51.103397    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:54:51.103982 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:54:51.103954    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:01.103609 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:01.103558    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:55:01.104021 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:01.104002    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:01.170743 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:01.170718    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 13:55:01.173089 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:01.173069    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 13:55:11.103715 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:11.103669    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:55:11.104285 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:11.104258    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:21.103760 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:21.103722    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:55:21.104252 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:21.104226    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:31.104732 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:31.104638    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:55:31.105218 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:31.104863    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:55:40.151371 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.151335    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"]
Jul 02 13:55:40.152196 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.152163    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" containerID="cri-o://7f672f841be14faafdcc4427bc04c530075e81a56ac40139cea25bc15d4b9a91" gracePeriod=30
Jul 02 13:55:40.152870 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.151667    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" containerID="cri-o://78ef739b566384e788353dd843a9b13fc460fef66c6f491fd78e3bf7bc8fdc11" gracePeriod=30
Jul 02 13:55:40.162511 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.162464    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy" containerID="cri-o://fc11010b8079ca5deed24e95f85b07fc0e8911c509a78c739df9748b2663be4a" gracePeriod=30
Jul 02 13:55:40.168096 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.168070    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"]
Jul 02 13:55:40.171374 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.171355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.173564 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.173544    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-8e592-kube-rbac-proxy-sar-config\""
Jul 02 13:55:40.173714 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.173697    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-8e592-predictor-serving-cert\""
Jul 02 13:55:40.182112 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.182092    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"]
Jul 02 13:55:40.242800 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.242777    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"]
Jul 02 13:55:40.245275 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.245261    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.249651 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.249632    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-8e592-kube-rbac-proxy-sar-config\""
Jul 02 13:55:40.249763 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.249671    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-8e592-predictor-serving-cert\""
Jul 02 13:55:40.259700 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.259683    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"]
Jul 02 13:55:40.324535 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67a4fdd-7894-447b-8301-28a285ee54e3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.324621 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-proxy-tls\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.324621 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.324621 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.324720 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245k5\" (UniqueName: \"kubernetes.io/projected/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-api-access-245k5\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.324720 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67a4fdd-7894-447b-8301-28a285ee54e3-proxy-tls\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.324788 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.324788 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.324898 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2cg\" (UniqueName: \"kubernetes.io/projected/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-api-access-qb2cg\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.324898 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.324815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.425406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67a4fdd-7894-447b-8301-28a285ee54e3-proxy-tls\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.425406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.425406 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.425598 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2cg\" (UniqueName: \"kubernetes.io/projected/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-api-access-qb2cg\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.425598 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.425598 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67a4fdd-7894-447b-8301-28a285ee54e3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.425598 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-proxy-tls\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.425785 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.425853 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425833    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.425923 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.425993 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.425973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245k5\" (UniqueName: \"kubernetes.io/projected/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-api-access-245k5\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.426304 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.426069    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67a4fdd-7894-447b-8301-28a285ee54e3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.426304 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.426249    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.426476 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.426312    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.426614 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.426592    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.426791 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.426772    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.427826 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.427807    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67a4fdd-7894-447b-8301-28a285ee54e3-proxy-tls\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.427951 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.427932    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-proxy-tls\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.435737 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.435710    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2cg\" (UniqueName: \"kubernetes.io/projected/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-api-access-qb2cg\") pod \"isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.436015 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.435997    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245k5\" (UniqueName: \"kubernetes.io/projected/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-api-access-245k5\") pod \"isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.446424 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.446398    2566 generic.go:356] "Generic (PLEG): container finished" podID="cb28b844-62f9-408a-8113-4168015658c1" containerID="fc11010b8079ca5deed24e95f85b07fc0e8911c509a78c739df9748b2663be4a" exitCode=2
Jul 02 13:55:40.446535 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.446467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerDied","Data":"fc11010b8079ca5deed24e95f85b07fc0e8911c509a78c739df9748b2663be4a"}
Jul 02 13:55:40.488389 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.488367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:40.555677 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.555647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:55:40.626080 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.626039    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"]
Jul 02 13:55:40.629672 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:55:40.629616    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67a4fdd_7894_447b_8301_28a285ee54e3.slice/crio-f8064d110f42a928c9e7abad96c563a5c1673035d76dc32a5e9d662aaf82c3f9 WatchSource:0}: Error finding container f8064d110f42a928c9e7abad96c563a5c1673035d76dc32a5e9d662aaf82c3f9: Status 404 returned error can't find the container with id f8064d110f42a928c9e7abad96c563a5c1673035d76dc32a5e9d662aaf82c3f9
Jul 02 13:55:40.698947 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:40.698863    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"]
Jul 02 13:55:40.700969 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:55:40.700945    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7478df_d9a6_4eaa_8128_8dd3254fe5b3.slice/crio-2ca3446c4e8a76d54440b4d3ffde73f56ecb687faf5fb070e164b6c2674ea279 WatchSource:0}: Error finding container 2ca3446c4e8a76d54440b4d3ffde73f56ecb687faf5fb070e164b6c2674ea279: Status 404 returned error can't find the container with id 2ca3446c4e8a76d54440b4d3ffde73f56ecb687faf5fb070e164b6c2674ea279
Jul 02 13:55:41.104190 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:41.104003    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:55:41.113806 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:41.104338    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:41.454058 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:41.454020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" event={"ID":"f67a4fdd-7894-447b-8301-28a285ee54e3","Type":"ContainerStarted","Data":"f87839755807bba1c475e0265f61a19c75cf9610107d09d67caef585b0e936c6"}
Jul 02 13:55:41.454467 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:41.454095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" event={"ID":"f67a4fdd-7894-447b-8301-28a285ee54e3","Type":"ContainerStarted","Data":"f8064d110f42a928c9e7abad96c563a5c1673035d76dc32a5e9d662aaf82c3f9"}
Jul 02 13:55:41.455723 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:41.455699    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" event={"ID":"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3","Type":"ContainerStarted","Data":"f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b"}
Jul 02 13:55:41.455821 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:41.455748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" event={"ID":"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3","Type":"ContainerStarted","Data":"2ca3446c4e8a76d54440b4d3ffde73f56ecb687faf5fb070e164b6c2674ea279"}
Jul 02 13:55:44.473973 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:44.473924    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerID="f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b" exitCode=0
Jul 02 13:55:44.474317 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:44.473983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" event={"ID":"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3","Type":"ContainerDied","Data":"f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b"}
Jul 02 13:55:44.476699 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:44.476681    2566 generic.go:356] "Generic (PLEG): container finished" podID="cb28b844-62f9-408a-8113-4168015658c1" containerID="7f672f841be14faafdcc4427bc04c530075e81a56ac40139cea25bc15d4b9a91" exitCode=0
Jul 02 13:55:44.476787 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:44.476754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerDied","Data":"7f672f841be14faafdcc4427bc04c530075e81a56ac40139cea25bc15d4b9a91"}
Jul 02 13:55:45.100570 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:45.100267    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 02 13:55:45.485869 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:45.485832    2566 generic.go:356] "Generic (PLEG): container finished" podID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerID="f87839755807bba1c475e0265f61a19c75cf9610107d09d67caef585b0e936c6" exitCode=0
Jul 02 13:55:45.486313 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:45.485920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" event={"ID":"f67a4fdd-7894-447b-8301-28a285ee54e3","Type":"ContainerDied","Data":"f87839755807bba1c475e0265f61a19c75cf9610107d09d67caef585b0e936c6"}
Jul 02 13:55:47.502455 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:47.502417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" event={"ID":"f67a4fdd-7894-447b-8301-28a285ee54e3","Type":"ContainerStarted","Data":"58181b3315597004148b509eed1f88b32e13adad4ceb808188bae76e26b12b4d"}
Jul 02 13:55:47.502940 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:47.502517    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" event={"ID":"f67a4fdd-7894-447b-8301-28a285ee54e3","Type":"ContainerStarted","Data":"9a27fcb8f64377f5f3a4592970eac016908e69d8aa338d3d4cf498913a04b0ca"}
Jul 02 13:55:47.502940 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:47.502721    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:47.526267 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:47.526218    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podStartSLOduration=3.663946708 podStartE2EDuration="7.526198819s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.862252111s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:55:47.523414706 +0000 UTC m=+647.365052219" watchObservedRunningTime="2026-07-02 13:55:47.526198819 +0000 UTC m=+647.367836334"
Jul 02 13:55:48.507432 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:48.507401    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:48.508646 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:48.508618    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:55:49.512638 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:49.512597    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:55:50.100408 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:50.100365    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 02 13:55:51.103565 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:51.103518    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:55:51.103989 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:51.103873    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:54.518170 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:54.518141    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:55:54.518629 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:54.518606    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:55:55.100085 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:55.100050    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 02 13:55:55.100253 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:55:55.100156    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:56:00.100922 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:00.100872    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 02 13:56:01.103765 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:01.103730    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 13:56:01.104252 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:01.103869    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:56:01.104252 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:01.104133    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:56:01.104356 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:01.104262    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:56:04.519094 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:04.519061    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:56:04.591257 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:04.591226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" event={"ID":"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3","Type":"ContainerStarted","Data":"76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5"}
Jul 02 13:56:04.591403 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:04.591294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" event={"ID":"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3","Type":"ContainerStarted","Data":"8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a"}
Jul 02 13:56:04.591542 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:04.591518    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:56:04.615272 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:04.615233    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podStartSLOduration=1.522721234 podStartE2EDuration="24.615219359s" totalImagesPullingTime="19.441320634s" totalInitContainerRuntime="3.651177491s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:56:04.613116697 +0000 UTC m=+664.454754211" watchObservedRunningTime="2026-07-02 13:56:04.615219359 +0000 UTC m=+664.456856872"
Jul 02 13:56:05.100039 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:05.100003    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 02 13:56:05.595967 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:05.595928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:56:05.597104 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:05.597080    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:56:06.600544 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:06.600501    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:56:10.100879 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.100835    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 02 13:56:10.620724 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.620694    2566 generic.go:356] "Generic (PLEG): container finished" podID="cb28b844-62f9-408a-8113-4168015658c1" containerID="78ef739b566384e788353dd843a9b13fc460fef66c6f491fd78e3bf7bc8fdc11" exitCode=0
Jul 02 13:56:10.620859 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.620763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerDied","Data":"78ef739b566384e788353dd843a9b13fc460fef66c6f491fd78e3bf7bc8fdc11"}
Jul 02 13:56:10.827629 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.827600    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:56:10.889048 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.889019    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb28b844-62f9-408a-8113-4168015658c1-proxy-tls\") pod \"cb28b844-62f9-408a-8113-4168015658c1\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") "
Jul 02 13:56:10.889171 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.889066    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-kube-rbac-proxy-sar-config\") pod \"cb28b844-62f9-408a-8113-4168015658c1\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") "
Jul 02 13:56:10.889171 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.889099    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb28b844-62f9-408a-8113-4168015658c1-kserve-provision-location\") pod \"cb28b844-62f9-408a-8113-4168015658c1\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") "
Jul 02 13:56:10.889285 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.889170    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szvmh\" (UniqueName: \"kubernetes.io/projected/cb28b844-62f9-408a-8113-4168015658c1-kube-api-access-szvmh\") pod \"cb28b844-62f9-408a-8113-4168015658c1\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") "
Jul 02 13:56:10.889285 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.889202    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-cabundle-cert\") pod \"cb28b844-62f9-408a-8113-4168015658c1\" (UID: \"cb28b844-62f9-408a-8113-4168015658c1\") "
Jul 02 13:56:10.889455 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.889432    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb28b844-62f9-408a-8113-4168015658c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb28b844-62f9-408a-8113-4168015658c1" (UID: "cb28b844-62f9-408a-8113-4168015658c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:10.889530 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.889404    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb28b844-62f9-408a-8113-4168015658c1" (UID: "cb28b844-62f9-408a-8113-4168015658c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:56:10.889604 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.889580    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb28b844-62f9-408a-8113-4168015658c1" (UID: "cb28b844-62f9-408a-8113-4168015658c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:56:10.891247 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.891229    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb28b844-62f9-408a-8113-4168015658c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb28b844-62f9-408a-8113-4168015658c1" (UID: "cb28b844-62f9-408a-8113-4168015658c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:56:10.891322 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.891269    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb28b844-62f9-408a-8113-4168015658c1-kube-api-access-szvmh" (OuterVolumeSpecName: "kube-api-access-szvmh") pod "cb28b844-62f9-408a-8113-4168015658c1" (UID: "cb28b844-62f9-408a-8113-4168015658c1"). InnerVolumeSpecName "kube-api-access-szvmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:56:10.990166 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.990141    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szvmh\" (UniqueName: \"kubernetes.io/projected/cb28b844-62f9-408a-8113-4168015658c1-kube-api-access-szvmh\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:56:10.990166 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.990165    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:56:10.990293 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.990175    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb28b844-62f9-408a-8113-4168015658c1-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:56:10.990293 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.990183    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb28b844-62f9-408a-8113-4168015658c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:56:10.990293 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:10.990192    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb28b844-62f9-408a-8113-4168015658c1-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:56:11.604462 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.604433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:56:11.604921 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.604896    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:56:11.627454 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.627430    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"
Jul 02 13:56:11.627454 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.627438    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf" event={"ID":"cb28b844-62f9-408a-8113-4168015658c1","Type":"ContainerDied","Data":"230762d6bdef92991d66875fb84aab4d40b3ca4411ce703661710f8540a391d7"}
Jul 02 13:56:11.627654 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.627514    2566 scope.go:117] "RemoveContainer" containerID="78ef739b566384e788353dd843a9b13fc460fef66c6f491fd78e3bf7bc8fdc11"
Jul 02 13:56:11.636782 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.636763    2566 scope.go:117] "RemoveContainer" containerID="fc11010b8079ca5deed24e95f85b07fc0e8911c509a78c739df9748b2663be4a"
Jul 02 13:56:11.644049 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.644034    2566 scope.go:117] "RemoveContainer" containerID="7f672f841be14faafdcc4427bc04c530075e81a56ac40139cea25bc15d4b9a91"
Jul 02 13:56:11.651661 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.651645    2566 scope.go:117] "RemoveContainer" containerID="53678a5220cbf5427c66a01b94ca6b385037a8c735605b1567fc295b5d960935"
Jul 02 13:56:11.661818 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.661798    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"]
Jul 02 13:56:11.671895 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:11.671868    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2cb97-predictor-75484bf457-px4pf"]
Jul 02 13:56:12.503293 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:12.503268    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb28b844-62f9-408a-8113-4168015658c1" path="/var/lib/kubelet/pods/cb28b844-62f9-408a-8113-4168015658c1/volumes"
Jul 02 13:56:14.519601 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:14.519568    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:56:21.605358 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:21.605321    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:56:24.518712 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:24.518681    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:56:31.605639 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:31.605598    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:56:34.519501 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:34.519451    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:56:41.605540 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:41.605505    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:56:44.518975 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:44.518934    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:56:51.604931 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:51.604891    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:56:54.519606 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:56:54.519525    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:57:01.605602 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:01.605535    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:57:04.520145 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:04.520109    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:57:30.620356 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.620313    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"]
Jul 02 13:57:30.621059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.620894    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="storage-initializer"
Jul 02 13:57:30.621059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.620915    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="storage-initializer"
Jul 02 13:57:30.621059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.620927    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container"
Jul 02 13:57:30.621059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.620938    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container"
Jul 02 13:57:30.621059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.620962    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent"
Jul 02 13:57:30.621059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.620970    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent"
Jul 02 13:57:30.621059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.620987    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy"
Jul 02 13:57:30.621059 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.620995    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy"
Jul 02 13:57:30.621561 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.621082    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="agent"
Jul 02 13:57:30.621561 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.621098    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kube-rbac-proxy"
Jul 02 13:57:30.621561 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.621107    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb28b844-62f9-408a-8113-4168015658c1" containerName="kserve-container"
Jul 02 13:57:30.623772 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.623748    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.626278 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.626258    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e16db-kube-rbac-proxy-sar-config\""
Jul 02 13:57:30.626884 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.626860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e16db-predictor-serving-cert\""
Jul 02 13:57:30.635661 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.635638    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"]
Jul 02 13:57:30.640542 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.640519    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"]
Jul 02 13:57:30.640826 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.640787    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" containerID="cri-o://9a27fcb8f64377f5f3a4592970eac016908e69d8aa338d3d4cf498913a04b0ca" gracePeriod=30
Jul 02 13:57:30.640907 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.640870    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kube-rbac-proxy" containerID="cri-o://58181b3315597004148b509eed1f88b32e13adad4ceb808188bae76e26b12b4d" gracePeriod=30
Jul 02 13:57:30.719403 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.719370    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"]
Jul 02 13:57:30.722102 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.722076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.724813 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.724790    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e16db-predictor-serving-cert\""
Jul 02 13:57:30.724987 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.724973    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e16db-kube-rbac-proxy-sar-config\""
Jul 02 13:57:30.729040 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.729017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/230a6f67-46d5-417c-a60e-12e235c73863-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.729128 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.729060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.729128 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.729097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8wt\" (UniqueName: \"kubernetes.io/projected/230a6f67-46d5-417c-a60e-12e235c73863-kube-api-access-rv8wt\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.729245 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.729199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230a6f67-46d5-417c-a60e-12e235c73863-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.729315 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.729287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.738017 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.737993    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"]
Jul 02 13:57:30.775142 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.775111    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"]
Jul 02 13:57:30.775352 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.775331    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container" containerID="cri-o://8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a" gracePeriod=30
Jul 02 13:57:30.775414 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.775366    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kube-rbac-proxy" containerID="cri-o://76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5" gracePeriod=30
Jul 02 13:57:30.830024 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.829994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/230a6f67-46d5-417c-a60e-12e235c73863-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.830127 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.830127 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8wt\" (UniqueName: \"kubernetes.io/projected/230a6f67-46d5-417c-a60e-12e235c73863-kube-api-access-rv8wt\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.830257 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c619fa6-4ad0-4ab9-8aea-338438880b09-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.830257 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.830368 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230a6f67-46d5-417c-a60e-12e235c73863-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.830368 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830345    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpws\" (UniqueName: \"kubernetes.io/projected/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-api-access-7vpws\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.830476 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c619fa6-4ad0-4ab9-8aea-338438880b09-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.830476 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.830476 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.830668 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830626    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230a6f67-46d5-417c-a60e-12e235c73863-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.830913 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.830890    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.831107 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.831077    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.832737 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.832713    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/230a6f67-46d5-417c-a60e-12e235c73863-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.839001 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.838977    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8wt\" (UniqueName: \"kubernetes.io/projected/230a6f67-46d5-417c-a60e-12e235c73863-kube-api-access-rv8wt\") pod \"isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.931686 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.931651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpws\" (UniqueName: \"kubernetes.io/projected/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-api-access-7vpws\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.931776 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.931693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c619fa6-4ad0-4ab9-8aea-338438880b09-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.931776 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.931719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.931857 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.931811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c619fa6-4ad0-4ab9-8aea-338438880b09-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.931857 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.931840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.932065 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.932035    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c619fa6-4ad0-4ab9-8aea-338438880b09-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.932448 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.932427    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.932556 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.932447    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.934334 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.934312    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c619fa6-4ad0-4ab9-8aea-338438880b09-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.936287 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.936269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:30.941955 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.941933    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpws\" (UniqueName: \"kubernetes.io/projected/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-api-access-7vpws\") pod \"isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:30.995777 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.995743    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerID="76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5" exitCode=2
Jul 02 13:57:30.995935 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.995816    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" event={"ID":"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3","Type":"ContainerDied","Data":"76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5"}
Jul 02 13:57:30.998770 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.998737    2566 generic.go:356] "Generic (PLEG): container finished" podID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerID="58181b3315597004148b509eed1f88b32e13adad4ceb808188bae76e26b12b4d" exitCode=2
Jul 02 13:57:30.998892 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:30.998781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" event={"ID":"f67a4fdd-7894-447b-8301-28a285ee54e3","Type":"ContainerDied","Data":"58181b3315597004148b509eed1f88b32e13adad4ceb808188bae76e26b12b4d"}
Jul 02 13:57:31.032890 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:31.032863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:31.080021 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:31.079996    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"]
Jul 02 13:57:31.083655 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:57:31.083628    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230a6f67_46d5_417c_a60e_12e235c73863.slice/crio-1c674d066cf1da9be401abd22eaa3cc659c93f4d46b8989cbfe9d039745d832d WatchSource:0}: Error finding container 1c674d066cf1da9be401abd22eaa3cc659c93f4d46b8989cbfe9d039745d832d: Status 404 returned error can't find the container with id 1c674d066cf1da9be401abd22eaa3cc659c93f4d46b8989cbfe9d039745d832d
Jul 02 13:57:31.177708 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:31.177682    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"]
Jul 02 13:57:31.179851 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:57:31.179825    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c619fa6_4ad0_4ab9_8aea_338438880b09.slice/crio-590d094b23fe75ec7c147fb2a04c07aea8369e9de87484c0bd90e9513eb95fb1 WatchSource:0}: Error finding container 590d094b23fe75ec7c147fb2a04c07aea8369e9de87484c0bd90e9513eb95fb1: Status 404 returned error can't find the container with id 590d094b23fe75ec7c147fb2a04c07aea8369e9de87484c0bd90e9513eb95fb1
Jul 02 13:57:31.601447 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:31.601400    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 02 13:57:31.605044 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:31.605016    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 13:57:32.011627 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:32.011584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" event={"ID":"230a6f67-46d5-417c-a60e-12e235c73863","Type":"ContainerStarted","Data":"dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c"}
Jul 02 13:57:32.012011 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:32.011690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" event={"ID":"230a6f67-46d5-417c-a60e-12e235c73863","Type":"ContainerStarted","Data":"1c674d066cf1da9be401abd22eaa3cc659c93f4d46b8989cbfe9d039745d832d"}
Jul 02 13:57:32.013051 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:32.013025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" event={"ID":"2c619fa6-4ad0-4ab9-8aea-338438880b09","Type":"ContainerStarted","Data":"9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516"}
Jul 02 13:57:32.013148 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:32.013075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" event={"ID":"2c619fa6-4ad0-4ab9-8aea-338438880b09","Type":"ContainerStarted","Data":"590d094b23fe75ec7c147fb2a04c07aea8369e9de87484c0bd90e9513eb95fb1"}
Jul 02 13:57:34.005917 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.005895    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:57:34.024873 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.024846    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerID="8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a" exitCode=0
Jul 02 13:57:34.024962 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.024893    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"
Jul 02 13:57:34.024962 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.024922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" event={"ID":"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3","Type":"ContainerDied","Data":"8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a"}
Jul 02 13:57:34.025041 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.024988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw" event={"ID":"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3","Type":"ContainerDied","Data":"2ca3446c4e8a76d54440b4d3ffde73f56ecb687faf5fb070e164b6c2674ea279"}
Jul 02 13:57:34.025041 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.025016    2566 scope.go:117] "RemoveContainer" containerID="76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5"
Jul 02 13:57:34.034350 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.034328    2566 scope.go:117] "RemoveContainer" containerID="8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a"
Jul 02 13:57:34.041964 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.041943    2566 scope.go:117] "RemoveContainer" containerID="f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b"
Jul 02 13:57:34.049048 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.049029    2566 scope.go:117] "RemoveContainer" containerID="76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5"
Jul 02 13:57:34.049331 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:57:34.049312    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5\": container with ID starting with 76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5 not found: ID does not exist" containerID="76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5"
Jul 02 13:57:34.049383 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.049340    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5"} err="failed to get container status \"76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5\": rpc error: code = NotFound desc = could not find container \"76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5\": container with ID starting with 76a1fa57aa7643d1266f8a1d8e06add6bcb7abf6ea21dc2e67bfb63696578ce5 not found: ID does not exist"
Jul 02 13:57:34.049383 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.049359    2566 scope.go:117] "RemoveContainer" containerID="8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a"
Jul 02 13:57:34.049624 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:57:34.049606    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a\": container with ID starting with 8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a not found: ID does not exist" containerID="8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a"
Jul 02 13:57:34.049683 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.049628    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a"} err="failed to get container status \"8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a\": rpc error: code = NotFound desc = could not find container \"8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a\": container with ID starting with 8be7927b31d76035133aa5e3832785f7222b031f409c898a8149076aaf70d30a not found: ID does not exist"
Jul 02 13:57:34.049683 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.049642    2566 scope.go:117] "RemoveContainer" containerID="f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b"
Jul 02 13:57:34.049885 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:57:34.049855    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b\": container with ID starting with f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b not found: ID does not exist" containerID="f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b"
Jul 02 13:57:34.049929 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.049893    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b"} err="failed to get container status \"f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b\": rpc error: code = NotFound desc = could not find container \"f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b\": container with ID starting with f45f89a548c1ef0769991d6284d13916ba745b6210feeff30b04193cdae4866b not found: ID does not exist"
Jul 02 13:57:34.158412 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158339    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-cabundle-cert\") pod \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") "
Jul 02 13:57:34.158412 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158403    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-rbac-proxy-sar-config\") pod \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") "
Jul 02 13:57:34.158643 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158458    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb2cg\" (UniqueName: \"kubernetes.io/projected/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-api-access-qb2cg\") pod \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") "
Jul 02 13:57:34.158643 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-proxy-tls\") pod \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") "
Jul 02 13:57:34.158643 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158533    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kserve-provision-location\") pod \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\" (UID: \"4b7478df-d9a6-4eaa-8128-8dd3254fe5b3\") "
Jul 02 13:57:34.158791 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158713    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" (UID: "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:57:34.158847 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158821    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" (UID: "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:57:34.158910 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158870    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" (UID: "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:57:34.158910 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158892    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:34.158910 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.158904    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:34.160588 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.160563    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-api-access-qb2cg" (OuterVolumeSpecName: "kube-api-access-qb2cg") pod "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" (UID: "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3"). InnerVolumeSpecName "kube-api-access-qb2cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:57:34.160702 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.160679    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" (UID: "4b7478df-d9a6-4eaa-8128-8dd3254fe5b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:57:34.259849 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.259821    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb2cg\" (UniqueName: \"kubernetes.io/projected/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kube-api-access-qb2cg\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:34.259849 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.259845    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:34.259849 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.259856    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:34.348476 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.348451    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"]
Jul 02 13:57:34.351139 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.351112    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8e592-predictor-75b67b8689-bs9jw"]
Jul 02 13:57:34.505308 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.505280    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" path="/var/lib/kubelet/pods/4b7478df-d9a6-4eaa-8128-8dd3254fe5b3/volumes"
Jul 02 13:57:34.513180 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.513151    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 02 13:57:34.518674 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:34.518648    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 13:57:35.031829 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.031766    2566 generic.go:356] "Generic (PLEG): container finished" podID="230a6f67-46d5-417c-a60e-12e235c73863" containerID="dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c" exitCode=0
Jul 02 13:57:35.031829 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.031806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" event={"ID":"230a6f67-46d5-417c-a60e-12e235c73863","Type":"ContainerDied","Data":"dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c"}
Jul 02 13:57:35.033265 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.033246    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerID="9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516" exitCode=0
Jul 02 13:57:35.033364 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.033322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" event={"ID":"2c619fa6-4ad0-4ab9-8aea-338438880b09","Type":"ContainerDied","Data":"9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516"}
Jul 02 13:57:35.035279 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.035262    2566 generic.go:356] "Generic (PLEG): container finished" podID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerID="9a27fcb8f64377f5f3a4592970eac016908e69d8aa338d3d4cf498913a04b0ca" exitCode=0
Jul 02 13:57:35.035367 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.035295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" event={"ID":"f67a4fdd-7894-447b-8301-28a285ee54e3","Type":"ContainerDied","Data":"9a27fcb8f64377f5f3a4592970eac016908e69d8aa338d3d4cf498913a04b0ca"}
Jul 02 13:57:35.035367 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.035328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h" event={"ID":"f67a4fdd-7894-447b-8301-28a285ee54e3","Type":"ContainerDied","Data":"f8064d110f42a928c9e7abad96c563a5c1673035d76dc32a5e9d662aaf82c3f9"}
Jul 02 13:57:35.035367 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.035350    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8064d110f42a928c9e7abad96c563a5c1673035d76dc32a5e9d662aaf82c3f9"
Jul 02 13:57:35.081124 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.081105    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:57:35.269134 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.269104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67a4fdd-7894-447b-8301-28a285ee54e3-proxy-tls\") pod \"f67a4fdd-7894-447b-8301-28a285ee54e3\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") "
Jul 02 13:57:35.269291 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.269205    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-cabundle-cert\") pod \"f67a4fdd-7894-447b-8301-28a285ee54e3\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") "
Jul 02 13:57:35.269291 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.269238    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67a4fdd-7894-447b-8301-28a285ee54e3-kserve-provision-location\") pod \"f67a4fdd-7894-447b-8301-28a285ee54e3\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") "
Jul 02 13:57:35.269422 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.269294    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-245k5\" (UniqueName: \"kubernetes.io/projected/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-api-access-245k5\") pod \"f67a4fdd-7894-447b-8301-28a285ee54e3\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") "
Jul 02 13:57:35.269422 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.269326    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-rbac-proxy-sar-config\") pod \"f67a4fdd-7894-447b-8301-28a285ee54e3\" (UID: \"f67a4fdd-7894-447b-8301-28a285ee54e3\") "
Jul 02 13:57:35.270127 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.270072    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f67a4fdd-7894-447b-8301-28a285ee54e3" (UID: "f67a4fdd-7894-447b-8301-28a285ee54e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:57:35.270127 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.270075    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67a4fdd-7894-447b-8301-28a285ee54e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f67a4fdd-7894-447b-8301-28a285ee54e3" (UID: "f67a4fdd-7894-447b-8301-28a285ee54e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:57:35.270421 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.270395    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f67a4fdd-7894-447b-8301-28a285ee54e3" (UID: "f67a4fdd-7894-447b-8301-28a285ee54e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:57:35.271929 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.271903    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67a4fdd-7894-447b-8301-28a285ee54e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f67a4fdd-7894-447b-8301-28a285ee54e3" (UID: "f67a4fdd-7894-447b-8301-28a285ee54e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:57:35.272225 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.272200    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-api-access-245k5" (OuterVolumeSpecName: "kube-api-access-245k5") pod "f67a4fdd-7894-447b-8301-28a285ee54e3" (UID: "f67a4fdd-7894-447b-8301-28a285ee54e3"). InnerVolumeSpecName "kube-api-access-245k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:57:35.370372 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.370286    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:35.370372 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.370324    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67a4fdd-7894-447b-8301-28a285ee54e3-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:35.370372 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.370340    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245k5\" (UniqueName: \"kubernetes.io/projected/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-api-access-245k5\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:35.370372 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.370355    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67a4fdd-7894-447b-8301-28a285ee54e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:35.370372 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:35.370373    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67a4fdd-7894-447b-8301-28a285ee54e3-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:57:36.041645 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.041612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" event={"ID":"230a6f67-46d5-417c-a60e-12e235c73863","Type":"ContainerStarted","Data":"f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225"}
Jul 02 13:57:36.042050 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.041705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" event={"ID":"230a6f67-46d5-417c-a60e-12e235c73863","Type":"ContainerStarted","Data":"3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c"}
Jul 02 13:57:36.042050 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.041901    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:36.043690 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.043665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" event={"ID":"2c619fa6-4ad0-4ab9-8aea-338438880b09","Type":"ContainerStarted","Data":"b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1"}
Jul 02 13:57:36.043784 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.043691    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"
Jul 02 13:57:36.043784 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.043716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" event={"ID":"2c619fa6-4ad0-4ab9-8aea-338438880b09","Type":"ContainerStarted","Data":"74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963"}
Jul 02 13:57:36.044085 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.044054    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:36.066041 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.066000    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podStartSLOduration=2.271390662 podStartE2EDuration="6.065987895s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794597233s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:57:36.064205382 +0000 UTC m=+755.905842896" watchObservedRunningTime="2026-07-02 13:57:36.065987895 +0000 UTC m=+755.907625407"
Jul 02 13:57:36.085644 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.085609    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podStartSLOduration=2.397804699 podStartE2EDuration="6.085597113s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.687792414s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:57:36.084076852 +0000 UTC m=+755.925714367" watchObservedRunningTime="2026-07-02 13:57:36.085597113 +0000 UTC m=+755.927234627"
Jul 02 13:57:36.097558 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.097540    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"]
Jul 02 13:57:36.102054 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.102032    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8e592-predictor-557bc657fc-lhd4h"]
Jul 02 13:57:36.504171 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:36.504140    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" path="/var/lib/kubelet/pods/f67a4fdd-7894-447b-8301-28a285ee54e3/volumes"
Jul 02 13:57:37.049287 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:37.049253    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:37.049715 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:37.049313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:37.050233 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:37.050206    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:57:37.050233 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:37.050221    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:57:38.054288 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:38.054247    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:57:38.054778 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:38.054356    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:57:43.059908 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:43.059879    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:57:43.060461 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:43.060431    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:57:43.061368 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:43.061347    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:57:43.061865 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:43.061842    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:57:53.060562 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:53.060525    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:57:53.062795 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:57:53.062772    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:58:03.061310 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:03.061274    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:58:03.061805 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:03.061754    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:58:13.061216 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:13.061176    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:58:13.062385 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:13.062363    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:58:23.060668 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:23.060578    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:58:23.061793 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:23.061766    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:58:33.061216 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:33.061184    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:58:33.061808 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:33.061780    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:58:43.061770 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:43.061736    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:58:53.062657 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:58:53.062628    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:59:20.907118 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907083    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"]
Jul 02 13:59:20.907702 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907659    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container"
Jul 02 13:59:20.907702 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907677    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container"
Jul 02 13:59:20.907702 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907692    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kube-rbac-proxy"
Jul 02 13:59:20.907702 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907701    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kube-rbac-proxy"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907723    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="storage-initializer"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907732    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="storage-initializer"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907746    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="storage-initializer"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907754    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="storage-initializer"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907763    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907771    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907799    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kube-rbac-proxy"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907808    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kube-rbac-proxy"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907897    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kube-rbac-proxy"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907914    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67a4fdd-7894-447b-8301-28a285ee54e3" containerName="kserve-container"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907924    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kube-rbac-proxy"
Jul 02 13:59:20.908004 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.907952    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b7478df-d9a6-4eaa-8128-8dd3254fe5b3" containerName="kserve-container"
Jul 02 13:59:20.910653 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.910631    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:20.913526 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.913476    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-1b7e6-predictor-serving-cert\""
Jul 02 13:59:20.913835 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.913817    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-1b7e6-kube-rbac-proxy-sar-config\""
Jul 02 13:59:20.923268 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.923246    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"]
Jul 02 13:59:20.957045 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.957020    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"]
Jul 02 13:59:20.957313 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.957285    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" containerID="cri-o://3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c" gracePeriod=30
Jul 02 13:59:20.957422 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:20.957309    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kube-rbac-proxy" containerID="cri-o://f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225" gracePeriod=30
Jul 02 13:59:21.001608 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.001588    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"]
Jul 02 13:59:21.001837 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.001812    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container" containerID="cri-o://74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963" gracePeriod=30
Jul 02 13:59:21.001929 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.001840    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kube-rbac-proxy" containerID="cri-o://b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1" gracePeriod=30
Jul 02 13:59:21.012359 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.012335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hb6g\" (UniqueName: \"kubernetes.io/projected/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-api-access-8hb6g\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.012456 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.012370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0558b3-ff74-4543-bd36-79ba2e659422-proxy-tls\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.012456 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.012415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.113570 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.113549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0558b3-ff74-4543-bd36-79ba2e659422-proxy-tls\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.113700 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.113683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.113785 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.113772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hb6g\" (UniqueName: \"kubernetes.io/projected/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-api-access-8hb6g\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.113844 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:21.113685    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-serving-cert: secret "message-dumper-raw-1b7e6-predictor-serving-cert" not found
Jul 02 13:59:21.113928 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:21.113875    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee0558b3-ff74-4543-bd36-79ba2e659422-proxy-tls podName:ee0558b3-ff74-4543-bd36-79ba2e659422 nodeName:}" failed. No retries permitted until 2026-07-02 13:59:21.613852652 +0000 UTC m=+861.455490156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ee0558b3-ff74-4543-bd36-79ba2e659422-proxy-tls") pod "message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" (UID: "ee0558b3-ff74-4543-bd36-79ba2e659422") : secret "message-dumper-raw-1b7e6-predictor-serving-cert" not found
Jul 02 13:59:21.114330 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.114306    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.121924 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.121904    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hb6g\" (UniqueName: \"kubernetes.io/projected/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-api-access-8hb6g\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.548675 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.548646    2566 generic.go:356] "Generic (PLEG): container finished" podID="230a6f67-46d5-417c-a60e-12e235c73863" containerID="f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225" exitCode=2
Jul 02 13:59:21.548843 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.548720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" event={"ID":"230a6f67-46d5-417c-a60e-12e235c73863","Type":"ContainerDied","Data":"f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225"}
Jul 02 13:59:21.550939 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.550920    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerID="b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1" exitCode=2
Jul 02 13:59:21.551052 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.550973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" event={"ID":"2c619fa6-4ad0-4ab9-8aea-338438880b09","Type":"ContainerDied","Data":"b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1"}
Jul 02 13:59:21.619289 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.619270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0558b3-ff74-4543-bd36-79ba2e659422-proxy-tls\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.621703 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.621688    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0558b3-ff74-4543-bd36-79ba2e659422-proxy-tls\") pod \"message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") " pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.824249 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.824176    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:21.952849 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.952827    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"]
Jul 02 13:59:21.955094 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:59:21.955072    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0558b3_ff74_4543_bd36_79ba2e659422.slice/crio-25818e62635bcf59e70c5a693d3bb0f1390d5cf241d8dcc5795a07836d8aab20 WatchSource:0}: Error finding container 25818e62635bcf59e70c5a693d3bb0f1390d5cf241d8dcc5795a07836d8aab20: Status 404 returned error can't find the container with id 25818e62635bcf59e70c5a693d3bb0f1390d5cf241d8dcc5795a07836d8aab20
Jul 02 13:59:21.956880 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:21.956862    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:59:22.556674 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:22.556640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" event={"ID":"ee0558b3-ff74-4543-bd36-79ba2e659422","Type":"ContainerStarted","Data":"25818e62635bcf59e70c5a693d3bb0f1390d5cf241d8dcc5795a07836d8aab20"}
Jul 02 13:59:23.054408 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:23.054363    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 02 13:59:23.054765 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:23.054446    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 02 13:59:23.060959 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:23.060936    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 13:59:23.061880 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:23.061856    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 13:59:23.562893 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:23.562850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" event={"ID":"ee0558b3-ff74-4543-bd36-79ba2e659422","Type":"ContainerStarted","Data":"571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95"}
Jul 02 13:59:23.563084 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:23.562917    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" event={"ID":"ee0558b3-ff74-4543-bd36-79ba2e659422","Type":"ContainerStarted","Data":"d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60"}
Jul 02 13:59:23.563084 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:23.563011    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:23.581995 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:23.581931    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" podStartSLOduration=2.377725808 podStartE2EDuration="3.581913664s" totalImagesPullingTime="1.204187856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:59:23.580637953 +0000 UTC m=+863.422275468" watchObservedRunningTime="2026-07-02 13:59:23.581913664 +0000 UTC m=+863.423551177"
Jul 02 13:59:24.346867 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.346844    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:59:24.441381 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.441359    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-rbac-proxy-sar-config\") pod \"2c619fa6-4ad0-4ab9-8aea-338438880b09\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") "
Jul 02 13:59:24.441502 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.441413    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-cabundle-cert\") pod \"2c619fa6-4ad0-4ab9-8aea-338438880b09\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") "
Jul 02 13:59:24.441502 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.441453    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c619fa6-4ad0-4ab9-8aea-338438880b09-proxy-tls\") pod \"2c619fa6-4ad0-4ab9-8aea-338438880b09\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") "
Jul 02 13:59:24.441502 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.441468    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c619fa6-4ad0-4ab9-8aea-338438880b09-kserve-provision-location\") pod \"2c619fa6-4ad0-4ab9-8aea-338438880b09\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") "
Jul 02 13:59:24.441671 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.441544    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vpws\" (UniqueName: \"kubernetes.io/projected/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-api-access-7vpws\") pod \"2c619fa6-4ad0-4ab9-8aea-338438880b09\" (UID: \"2c619fa6-4ad0-4ab9-8aea-338438880b09\") "
Jul 02 13:59:24.441838 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.441714    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c619fa6-4ad0-4ab9-8aea-338438880b09" (UID: "2c619fa6-4ad0-4ab9-8aea-338438880b09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:59:24.441898 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.441825    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c619fa6-4ad0-4ab9-8aea-338438880b09" (UID: "2c619fa6-4ad0-4ab9-8aea-338438880b09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:59:24.441898 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.441857    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c619fa6-4ad0-4ab9-8aea-338438880b09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c619fa6-4ad0-4ab9-8aea-338438880b09" (UID: "2c619fa6-4ad0-4ab9-8aea-338438880b09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:59:24.443411 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.443392    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c619fa6-4ad0-4ab9-8aea-338438880b09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c619fa6-4ad0-4ab9-8aea-338438880b09" (UID: "2c619fa6-4ad0-4ab9-8aea-338438880b09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:59:24.443515 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.443473    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-api-access-7vpws" (OuterVolumeSpecName: "kube-api-access-7vpws") pod "2c619fa6-4ad0-4ab9-8aea-338438880b09" (UID: "2c619fa6-4ad0-4ab9-8aea-338438880b09"). InnerVolumeSpecName "kube-api-access-7vpws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:59:24.542629 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.542609    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:24.542629 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.542628    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c619fa6-4ad0-4ab9-8aea-338438880b09-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:24.542740 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.542638    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c619fa6-4ad0-4ab9-8aea-338438880b09-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:24.542740 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.542646    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c619fa6-4ad0-4ab9-8aea-338438880b09-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:24.542740 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.542654    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vpws\" (UniqueName: \"kubernetes.io/projected/2c619fa6-4ad0-4ab9-8aea-338438880b09-kube-api-access-7vpws\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:24.569792 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.569765    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerID="74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963" exitCode=0
Jul 02 13:59:24.569892 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.569823    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"
Jul 02 13:59:24.569892 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.569810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" event={"ID":"2c619fa6-4ad0-4ab9-8aea-338438880b09","Type":"ContainerDied","Data":"74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963"}
Jul 02 13:59:24.569892 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.569889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w" event={"ID":"2c619fa6-4ad0-4ab9-8aea-338438880b09","Type":"ContainerDied","Data":"590d094b23fe75ec7c147fb2a04c07aea8369e9de87484c0bd90e9513eb95fb1"}
Jul 02 13:59:24.570060 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.569929    2566 scope.go:117] "RemoveContainer" containerID="b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1"
Jul 02 13:59:24.573549 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.571361    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:24.577789 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.577772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:24.584983 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.584964    2566 scope.go:117] "RemoveContainer" containerID="74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963"
Jul 02 13:59:24.592155 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.592135    2566 scope.go:117] "RemoveContainer" containerID="9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516"
Jul 02 13:59:24.599264 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.599245    2566 scope.go:117] "RemoveContainer" containerID="b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1"
Jul 02 13:59:24.599687 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:24.599662    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1\": container with ID starting with b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1 not found: ID does not exist" containerID="b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1"
Jul 02 13:59:24.599781 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.599726    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1"} err="failed to get container status \"b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1\": rpc error: code = NotFound desc = could not find container \"b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1\": container with ID starting with b539f1c6473a2b66c2c1ee0797f1e8720d362070b0d09480691d98865e8976b1 not found: ID does not exist"
Jul 02 13:59:24.599781 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.599751    2566 scope.go:117] "RemoveContainer" containerID="74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963"
Jul 02 13:59:24.600029 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:24.600012    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963\": container with ID starting with 74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963 not found: ID does not exist" containerID="74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963"
Jul 02 13:59:24.600097 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.600037    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963"} err="failed to get container status \"74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963\": rpc error: code = NotFound desc = could not find container \"74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963\": container with ID starting with 74785f0b54409e8cb15089b4231b32f978836e0546db357bbe5087f38d97c963 not found: ID does not exist"
Jul 02 13:59:24.600097 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.600057    2566 scope.go:117] "RemoveContainer" containerID="9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516"
Jul 02 13:59:24.600298 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:24.600280    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516\": container with ID starting with 9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516 not found: ID does not exist" containerID="9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516"
Jul 02 13:59:24.600336 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.600304    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516"} err="failed to get container status \"9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516\": rpc error: code = NotFound desc = could not find container \"9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516\": container with ID starting with 9ef0338d06d56ce367eb4c164d103e78506e54929d697e5a33586669656ef516 not found: ID does not exist"
Jul 02 13:59:24.614137 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.614114    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"]
Jul 02 13:59:24.619393 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:24.619372    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e16db-predictor-666d58894f-6fr2w"]
Jul 02 13:59:25.403356 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.403330    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:59:25.551359 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.551334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv8wt\" (UniqueName: \"kubernetes.io/projected/230a6f67-46d5-417c-a60e-12e235c73863-kube-api-access-rv8wt\") pod \"230a6f67-46d5-417c-a60e-12e235c73863\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") "
Jul 02 13:59:25.551538 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.551396    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/230a6f67-46d5-417c-a60e-12e235c73863-proxy-tls\") pod \"230a6f67-46d5-417c-a60e-12e235c73863\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") "
Jul 02 13:59:25.551538 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.551426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-kube-rbac-proxy-sar-config\") pod \"230a6f67-46d5-417c-a60e-12e235c73863\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") "
Jul 02 13:59:25.551538 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.551476    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230a6f67-46d5-417c-a60e-12e235c73863-kserve-provision-location\") pod \"230a6f67-46d5-417c-a60e-12e235c73863\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") "
Jul 02 13:59:25.551538 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.551524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-cabundle-cert\") pod \"230a6f67-46d5-417c-a60e-12e235c73863\" (UID: \"230a6f67-46d5-417c-a60e-12e235c73863\") "
Jul 02 13:59:25.551773 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.551747    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "230a6f67-46d5-417c-a60e-12e235c73863" (UID: "230a6f67-46d5-417c-a60e-12e235c73863"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:59:25.551866 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.551843    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230a6f67-46d5-417c-a60e-12e235c73863-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "230a6f67-46d5-417c-a60e-12e235c73863" (UID: "230a6f67-46d5-417c-a60e-12e235c73863"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:59:25.551929 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.551905    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "230a6f67-46d5-417c-a60e-12e235c73863" (UID: "230a6f67-46d5-417c-a60e-12e235c73863"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:59:25.553399 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.553376    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230a6f67-46d5-417c-a60e-12e235c73863-kube-api-access-rv8wt" (OuterVolumeSpecName: "kube-api-access-rv8wt") pod "230a6f67-46d5-417c-a60e-12e235c73863" (UID: "230a6f67-46d5-417c-a60e-12e235c73863"). InnerVolumeSpecName "kube-api-access-rv8wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:59:25.553607 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.553590    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230a6f67-46d5-417c-a60e-12e235c73863-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "230a6f67-46d5-417c-a60e-12e235c73863" (UID: "230a6f67-46d5-417c-a60e-12e235c73863"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:59:25.577865 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.577843    2566 generic.go:356] "Generic (PLEG): container finished" podID="230a6f67-46d5-417c-a60e-12e235c73863" containerID="3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c" exitCode=0
Jul 02 13:59:25.577949 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.577897    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"
Jul 02 13:59:25.578002 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.577938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" event={"ID":"230a6f67-46d5-417c-a60e-12e235c73863","Type":"ContainerDied","Data":"3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c"}
Jul 02 13:59:25.578041 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.578007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv" event={"ID":"230a6f67-46d5-417c-a60e-12e235c73863","Type":"ContainerDied","Data":"1c674d066cf1da9be401abd22eaa3cc659c93f4d46b8989cbfe9d039745d832d"}
Jul 02 13:59:25.578041 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.578035    2566 scope.go:117] "RemoveContainer" containerID="f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225"
Jul 02 13:59:25.586415 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.586399    2566 scope.go:117] "RemoveContainer" containerID="3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c"
Jul 02 13:59:25.593750 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.593729    2566 scope.go:117] "RemoveContainer" containerID="dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c"
Jul 02 13:59:25.601833 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.601817    2566 scope.go:117] "RemoveContainer" containerID="f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225"
Jul 02 13:59:25.601890 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.601825    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"]
Jul 02 13:59:25.602089 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:25.602073    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225\": container with ID starting with f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225 not found: ID does not exist" containerID="f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225"
Jul 02 13:59:25.602128 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.602096    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225"} err="failed to get container status \"f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225\": rpc error: code = NotFound desc = could not find container \"f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225\": container with ID starting with f8ef815f14f56ee8982724675e45bcafe063e53c22a149535aea0add75c52225 not found: ID does not exist"
Jul 02 13:59:25.602128 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.602111    2566 scope.go:117] "RemoveContainer" containerID="3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c"
Jul 02 13:59:25.602327 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:25.602310    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c\": container with ID starting with 3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c not found: ID does not exist" containerID="3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c"
Jul 02 13:59:25.602362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.602333    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c"} err="failed to get container status \"3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c\": rpc error: code = NotFound desc = could not find container \"3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c\": container with ID starting with 3920bf77526f6f963d346c8a110a6fa6919bea09c0c5643df4e5bd3d85d6a16c not found: ID does not exist"
Jul 02 13:59:25.602362 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.602348    2566 scope.go:117] "RemoveContainer" containerID="dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c"
Jul 02 13:59:25.602580 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:25.602556    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c\": container with ID starting with dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c not found: ID does not exist" containerID="dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c"
Jul 02 13:59:25.602622 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.602586    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c"} err="failed to get container status \"dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c\": rpc error: code = NotFound desc = could not find container \"dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c\": container with ID starting with dbe209717415187e3907896a5f6cb069c220269c20268100254c38baf6308a8c not found: ID does not exist"
Jul 02 13:59:25.606767 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.606750    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e16db-predictor-7b7f64c5c7-fg2sv"]
Jul 02 13:59:25.652743 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.652719    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv8wt\" (UniqueName: \"kubernetes.io/projected/230a6f67-46d5-417c-a60e-12e235c73863-kube-api-access-rv8wt\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:25.652743 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.652741    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/230a6f67-46d5-417c-a60e-12e235c73863-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:25.652891 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.652753    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:25.652891 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.652763    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230a6f67-46d5-417c-a60e-12e235c73863-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:25.652891 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:25.652773    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230a6f67-46d5-417c-a60e-12e235c73863-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 13:59:26.504270 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:26.504239    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230a6f67-46d5-417c-a60e-12e235c73863" path="/var/lib/kubelet/pods/230a6f67-46d5-417c-a60e-12e235c73863/volumes"
Jul 02 13:59:26.504755 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:26.504739    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" path="/var/lib/kubelet/pods/2c619fa6-4ad0-4ab9-8aea-338438880b09/volumes"
Jul 02 13:59:31.587459 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:31.587430    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 13:59:40.930826 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.930792    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"]
Jul 02 13:59:40.931171 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931146    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="storage-initializer"
Jul 02 13:59:40.931171 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931158    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="storage-initializer"
Jul 02 13:59:40.931171 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931171    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="storage-initializer"
Jul 02 13:59:40.931284 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931178    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="storage-initializer"
Jul 02 13:59:40.931284 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931188    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container"
Jul 02 13:59:40.931284 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931194    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container"
Jul 02 13:59:40.931284 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931201    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kube-rbac-proxy"
Jul 02 13:59:40.931284 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931208    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kube-rbac-proxy"
Jul 02 13:59:40.931284 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931217    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container"
Jul 02 13:59:40.931284 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931222    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container"
Jul 02 13:59:40.931284 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931228    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kube-rbac-proxy"
Jul 02 13:59:40.931284 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931233    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kube-rbac-proxy"
Jul 02 13:59:40.931556 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931296    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kserve-container"
Jul 02 13:59:40.931556 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931310    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kube-rbac-proxy"
Jul 02 13:59:40.931556 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931320    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c619fa6-4ad0-4ab9-8aea-338438880b09" containerName="kube-rbac-proxy"
Jul 02 13:59:40.931556 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.931332    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="230a6f67-46d5-417c-a60e-12e235c73863" containerName="kserve-container"
Jul 02 13:59:40.934271 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.934253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:40.936645 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.936621    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:59:40.936645 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.936645    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 13:59:40.936807 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.936648    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-1b7e6-kube-rbac-proxy-sar-config\""
Jul 02 13:59:40.936807 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.936621    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-1b7e6-predictor-serving-cert\""
Jul 02 13:59:40.947427 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.947408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"]
Jul 02 13:59:40.963842 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.963817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kserve-provision-location\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:40.963939 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.963848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:40.963939 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.963867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54z9x\" (UniqueName: \"kubernetes.io/projected/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-api-access-54z9x\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:40.963939 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.963891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-cabundle-cert\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:40.963939 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:40.963919    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.064527 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.064496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54z9x\" (UniqueName: \"kubernetes.io/projected/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-api-access-54z9x\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.064635 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.064542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-cabundle-cert\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.064635 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.064572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.064790 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.064775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kserve-provision-location\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.064845 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.064816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.064845 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:41.064782    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-serving-cert: secret "isvc-logger-raw-1b7e6-predictor-serving-cert" not found
Jul 02 13:59:41.064950 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:41.064912    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls podName:d195ac35-8371-4f55-96a1-f3e8b5c785a5 nodeName:}" failed. No retries permitted until 2026-07-02 13:59:41.564889187 +0000 UTC m=+881.406526681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls") pod "isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" (UID: "d195ac35-8371-4f55-96a1-f3e8b5c785a5") : secret "isvc-logger-raw-1b7e6-predictor-serving-cert" not found
Jul 02 13:59:41.065102 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.065078    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kserve-provision-location\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.065267 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.065248    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-cabundle-cert\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.065389 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.065371    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.073577 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.073553    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54z9x\" (UniqueName: \"kubernetes.io/projected/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-api-access-54z9x\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.569441 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:41.569415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:41.569567 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:41.569552    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-serving-cert: secret "isvc-logger-raw-1b7e6-predictor-serving-cert" not found
Jul 02 13:59:41.569616 ip-10-0-136-2 kubenswrapper[2566]: E0702 13:59:41.569608    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls podName:d195ac35-8371-4f55-96a1-f3e8b5c785a5 nodeName:}" failed. No retries permitted until 2026-07-02 13:59:42.569596303 +0000 UTC m=+882.411233794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls") pod "isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" (UID: "d195ac35-8371-4f55-96a1-f3e8b5c785a5") : secret "isvc-logger-raw-1b7e6-predictor-serving-cert" not found
Jul 02 13:59:42.577165 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:42.577126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:42.584644 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:42.584607    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls\") pod \"isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:42.744029 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:42.744000    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:42.882080 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:42.882057    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"]
Jul 02 13:59:42.884721 ip-10-0-136-2 kubenswrapper[2566]: W0702 13:59:42.884692    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd195ac35_8371_4f55_96a1_f3e8b5c785a5.slice/crio-c2f0d14850a820dd503c63a029ad311fb9bf815dc4fc72db1d7975e45c8507f6 WatchSource:0}: Error finding container c2f0d14850a820dd503c63a029ad311fb9bf815dc4fc72db1d7975e45c8507f6: Status 404 returned error can't find the container with id c2f0d14850a820dd503c63a029ad311fb9bf815dc4fc72db1d7975e45c8507f6
Jul 02 13:59:43.661964 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:43.661929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerStarted","Data":"b434e14902d9499303c41677bc4cc3e89bfd202bfedff71598c8d3d083b26d6b"}
Jul 02 13:59:43.662428 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:43.662006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerStarted","Data":"c2f0d14850a820dd503c63a029ad311fb9bf815dc4fc72db1d7975e45c8507f6"}
Jul 02 13:59:47.681625 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:47.681590    2566 generic.go:356] "Generic (PLEG): container finished" podID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerID="b434e14902d9499303c41677bc4cc3e89bfd202bfedff71598c8d3d083b26d6b" exitCode=0
Jul 02 13:59:47.682045 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:47.681645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerDied","Data":"b434e14902d9499303c41677bc4cc3e89bfd202bfedff71598c8d3d083b26d6b"}
Jul 02 13:59:48.690143 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:48.690108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerStarted","Data":"94ea480f3c77164b9e886993bc277eb7d1a0de32bdfe1a744ece2962756b4c16"}
Jul 02 13:59:48.690579 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:48.690176    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerStarted","Data":"0c922f3edcca09b1d4b2f1b85c5babf6b5bf81a77e048888e5efe9fedbdd89b6"}
Jul 02 13:59:48.690579 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:48.690200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerStarted","Data":"4b54ce651c541f0508052d2f046f7761e2134540ba7f194f96fb1cfc751ed0ef"}
Jul 02 13:59:48.690579 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:48.690494    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:48.690684 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:48.690641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:48.691949 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:48.691926    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:59:48.715562 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:48.715522    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podStartSLOduration=4.914381275 podStartE2EDuration="8.715510565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.80112929s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:59:48.713506071 +0000 UTC m=+888.555143581" watchObservedRunningTime="2026-07-02 13:59:48.715510565 +0000 UTC m=+888.557148079"
Jul 02 13:59:49.696013 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:49.695986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:49.696371 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:49.696074    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:59:49.697171 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:49.697146    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:59:50.700379 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:50.700338    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:59:50.700831 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:50.700806    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:59:55.705542 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:55.705449    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 13:59:55.706034 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:55.706009    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 13:59:55.706331 ip-10-0-136-2 kubenswrapper[2566]: I0702 13:59:55.706314    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:00:01.204817 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:01.204784    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 14:00:01.205302 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:01.204886    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 14:00:05.706765 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:05.706721    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:00:05.707176 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:05.707135    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:00:15.706415 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:15.706375    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:00:15.706888 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:15.706822    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:00:25.706192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:25.706148    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:00:25.706670 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:25.706643    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:00:35.706090 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:35.706043    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:00:35.706576 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:35.706504    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:00:45.706727 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:45.706680    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:00:45.708742 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:45.707165    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:00:55.706736 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:55.706686    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:00:55.707230 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:00:55.707153    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:01:05.706711 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:05.706678    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 14:01:05.707184 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:05.706776    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 14:01:15.968994 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:15.968962    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v_ee0558b3-ff74-4543-bd36-79ba2e659422/kserve-container/0.log"
Jul 02 14:01:16.186034 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.185997    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"]
Jul 02 14:01:16.186294 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.186264    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" containerID="cri-o://4b54ce651c541f0508052d2f046f7761e2134540ba7f194f96fb1cfc751ed0ef" gracePeriod=30
Jul 02 14:01:16.186450 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.186308    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" containerID="cri-o://94ea480f3c77164b9e886993bc277eb7d1a0de32bdfe1a744ece2962756b4c16" gracePeriod=30
Jul 02 14:01:16.186585 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.186328    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy" containerID="cri-o://0c922f3edcca09b1d4b2f1b85c5babf6b5bf81a77e048888e5efe9fedbdd89b6" gracePeriod=30
Jul 02 14:01:16.269119 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.269087    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"]
Jul 02 14:01:16.272899 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.272880    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.275623 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.275602    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-1c082-predictor-serving-cert\""
Jul 02 14:01:16.275900 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.275884    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-1c082-kube-rbac-proxy-sar-config\""
Jul 02 14:01:16.285587 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.285563    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"]
Jul 02 14:01:16.307282 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.307257    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"]
Jul 02 14:01:16.307741 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.307691    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" podUID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerName="kserve-container" containerID="cri-o://d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60" gracePeriod=30
Jul 02 14:01:16.308216 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.308168    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" podUID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerName="kube-rbac-proxy" containerID="cri-o://571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95" gracePeriod=30
Jul 02 14:01:16.462511 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.462330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-proxy-tls\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.462511 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.462377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.462511 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.462426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.462511 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.462447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.462511 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.462470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpmbz\" (UniqueName: \"kubernetes.io/projected/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-api-access-lpmbz\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.558221 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.558197    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 14:01:16.563046 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.563026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-proxy-tls\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.563141 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.563060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.563141 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.563088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.563141 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.563114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.563141 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.563130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpmbz\" (UniqueName: \"kubernetes.io/projected/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-api-access-lpmbz\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.563469 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.563447    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.563751 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.563723    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.563845 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.563781    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.565529 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.565512    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-proxy-tls\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.570751 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.570732    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpmbz\" (UniqueName: \"kubernetes.io/projected/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-api-access-lpmbz\") pod \"isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.586094 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.586068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:16.664025 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.663992    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hb6g\" (UniqueName: \"kubernetes.io/projected/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-api-access-8hb6g\") pod \"ee0558b3-ff74-4543-bd36-79ba2e659422\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") "
Jul 02 14:01:16.664194 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.664047    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0558b3-ff74-4543-bd36-79ba2e659422-proxy-tls\") pod \"ee0558b3-ff74-4543-bd36-79ba2e659422\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") "
Jul 02 14:01:16.664194 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.664096    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-rbac-proxy-sar-config\") pod \"ee0558b3-ff74-4543-bd36-79ba2e659422\" (UID: \"ee0558b3-ff74-4543-bd36-79ba2e659422\") "
Jul 02 14:01:16.664567 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.664541    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee0558b3-ff74-4543-bd36-79ba2e659422" (UID: "ee0558b3-ff74-4543-bd36-79ba2e659422"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:01:16.666019 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.665983    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0558b3-ff74-4543-bd36-79ba2e659422-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee0558b3-ff74-4543-bd36-79ba2e659422" (UID: "ee0558b3-ff74-4543-bd36-79ba2e659422"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:01:16.666019 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.665995    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-api-access-8hb6g" (OuterVolumeSpecName: "kube-api-access-8hb6g") pod "ee0558b3-ff74-4543-bd36-79ba2e659422" (UID: "ee0558b3-ff74-4543-bd36-79ba2e659422"). InnerVolumeSpecName "kube-api-access-8hb6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:01:16.719558 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.719533    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"]
Jul 02 14:01:16.721801 ip-10-0-136-2 kubenswrapper[2566]: W0702 14:01:16.721772    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc540ec_5678_4ea3_8a6b_5f8ffc43ecd3.slice/crio-295affae33f84cdeb465b1e3590e4ea867ed96a58e5a9d40756e6c7c7fc67936 WatchSource:0}: Error finding container 295affae33f84cdeb465b1e3590e4ea867ed96a58e5a9d40756e6c7c7fc67936: Status 404 returned error can't find the container with id 295affae33f84cdeb465b1e3590e4ea867ed96a58e5a9d40756e6c7c7fc67936
Jul 02 14:01:16.764801 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.764778    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hb6g\" (UniqueName: \"kubernetes.io/projected/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-api-access-8hb6g\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:01:16.764801 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.764801    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0558b3-ff74-4543-bd36-79ba2e659422-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:01:16.764938 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:16.764812    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0558b3-ff74-4543-bd36-79ba2e659422-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:01:17.126168 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.126085    2566 generic.go:356] "Generic (PLEG): container finished" podID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerID="0c922f3edcca09b1d4b2f1b85c5babf6b5bf81a77e048888e5efe9fedbdd89b6" exitCode=2
Jul 02 14:01:17.126571 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.126161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerDied","Data":"0c922f3edcca09b1d4b2f1b85c5babf6b5bf81a77e048888e5efe9fedbdd89b6"}
Jul 02 14:01:17.127743 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.127718    2566 generic.go:356] "Generic (PLEG): container finished" podID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerID="571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95" exitCode=2
Jul 02 14:01:17.127743 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.127739    2566 generic.go:356] "Generic (PLEG): container finished" podID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerID="d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60" exitCode=2
Jul 02 14:01:17.127928 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.127778    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"
Jul 02 14:01:17.127928 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.127799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" event={"ID":"ee0558b3-ff74-4543-bd36-79ba2e659422","Type":"ContainerDied","Data":"571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95"}
Jul 02 14:01:17.127928 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.127849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" event={"ID":"ee0558b3-ff74-4543-bd36-79ba2e659422","Type":"ContainerDied","Data":"d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60"}
Jul 02 14:01:17.127928 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.127877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v" event={"ID":"ee0558b3-ff74-4543-bd36-79ba2e659422","Type":"ContainerDied","Data":"25818e62635bcf59e70c5a693d3bb0f1390d5cf241d8dcc5795a07836d8aab20"}
Jul 02 14:01:17.127928 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.127909    2566 scope.go:117] "RemoveContainer" containerID="571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95"
Jul 02 14:01:17.129459 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.129433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" event={"ID":"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3","Type":"ContainerStarted","Data":"9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a"}
Jul 02 14:01:17.129567 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.129521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" event={"ID":"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3","Type":"ContainerStarted","Data":"295affae33f84cdeb465b1e3590e4ea867ed96a58e5a9d40756e6c7c7fc67936"}
Jul 02 14:01:17.141178 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.141156    2566 scope.go:117] "RemoveContainer" containerID="d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60"
Jul 02 14:01:17.150417 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.150394    2566 scope.go:117] "RemoveContainer" containerID="571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95"
Jul 02 14:01:17.150713 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:01:17.150695    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95\": container with ID starting with 571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95 not found: ID does not exist" containerID="571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95"
Jul 02 14:01:17.150774 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.150721    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95"} err="failed to get container status \"571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95\": rpc error: code = NotFound desc = could not find container \"571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95\": container with ID starting with 571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95 not found: ID does not exist"
Jul 02 14:01:17.150774 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.150738    2566 scope.go:117] "RemoveContainer" containerID="d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60"
Jul 02 14:01:17.151017 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:01:17.150998    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60\": container with ID starting with d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60 not found: ID does not exist" containerID="d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60"
Jul 02 14:01:17.151092 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.151027    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60"} err="failed to get container status \"d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60\": rpc error: code = NotFound desc = could not find container \"d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60\": container with ID starting with d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60 not found: ID does not exist"
Jul 02 14:01:17.151092 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.151049    2566 scope.go:117] "RemoveContainer" containerID="571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95"
Jul 02 14:01:17.151290 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.151272    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95"} err="failed to get container status \"571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95\": rpc error: code = NotFound desc = could not find container \"571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95\": container with ID starting with 571f1abe17c47cfcbe7aeab969c3f8e31bb24393cd15a46608a91d5ac6076f95 not found: ID does not exist"
Jul 02 14:01:17.151336 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.151292    2566 scope.go:117] "RemoveContainer" containerID="d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60"
Jul 02 14:01:17.151513 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.151474    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60"} err="failed to get container status \"d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60\": rpc error: code = NotFound desc = could not find container \"d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60\": container with ID starting with d69fdf4fd047b7f07a808132d4990ec0c701f9c2f5620d30e751b4ce1ea11a60 not found: ID does not exist"
Jul 02 14:01:17.215925 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.215894    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"]
Jul 02 14:01:17.234675 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:17.234642    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-1b7e6-predictor-64df5c9c95-62h4v"]
Jul 02 14:01:18.503107 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:18.503077    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee0558b3-ff74-4543-bd36-79ba2e659422" path="/var/lib/kubelet/pods/ee0558b3-ff74-4543-bd36-79ba2e659422/volumes"
Jul 02 14:01:20.700688 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:20.700659    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 14:01:21.155499 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:21.155447    2566 generic.go:356] "Generic (PLEG): container finished" podID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerID="4b54ce651c541f0508052d2f046f7761e2134540ba7f194f96fb1cfc751ed0ef" exitCode=0
Jul 02 14:01:21.155684 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:21.155519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerDied","Data":"4b54ce651c541f0508052d2f046f7761e2134540ba7f194f96fb1cfc751ed0ef"}
Jul 02 14:01:21.156803 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:21.156782    2566 generic.go:356] "Generic (PLEG): container finished" podID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerID="9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a" exitCode=0
Jul 02 14:01:21.156910 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:21.156811    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" event={"ID":"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3","Type":"ContainerDied","Data":"9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a"}
Jul 02 14:01:22.164047 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:22.164008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" event={"ID":"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3","Type":"ContainerStarted","Data":"bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46"}
Jul 02 14:01:22.164580 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:22.164083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" event={"ID":"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3","Type":"ContainerStarted","Data":"52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b"}
Jul 02 14:01:22.164580 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:22.164405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:22.184305 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:22.184257    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podStartSLOduration=2.552795528 podStartE2EDuration="6.184241767s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.631446239s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:01:22.182895867 +0000 UTC m=+982.024533385" watchObservedRunningTime="2026-07-02 14:01:22.184241767 +0000 UTC m=+982.025879281"
Jul 02 14:01:23.168697 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:23.168663    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:23.169975 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:23.169936    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:01:24.173613 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:24.173571    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:01:25.700440 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:25.700400    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 14:01:25.705948 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:25.705899    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:01:25.706217 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:25.706200    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:01:29.178075 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:29.178044    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:01:29.178565 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:29.178539    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:01:30.701074 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:30.701030    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 14:01:30.701475 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:30.701158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 14:01:35.700756 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:35.700715    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 14:01:35.706063 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:35.706037    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:01:35.706473 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:35.706453    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:01:39.179183 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:39.179146    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:01:40.700800 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:40.700761    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 14:01:45.701125 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:45.701079    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 14:01:45.706712 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:45.706673    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:01:45.706835 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:45.706804    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 14:01:45.707033 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:45.706990    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:01:45.707100 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:45.707088    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 14:01:46.284159 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.284114    2566 generic.go:356] "Generic (PLEG): container finished" podID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerID="94ea480f3c77164b9e886993bc277eb7d1a0de32bdfe1a744ece2962756b4c16" exitCode=0
Jul 02 14:01:46.284362 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.284188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerDied","Data":"94ea480f3c77164b9e886993bc277eb7d1a0de32bdfe1a744ece2962756b4c16"}
Jul 02 14:01:46.834547 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.834523    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 14:01:46.930463 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.930428    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54z9x\" (UniqueName: \"kubernetes.io/projected/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-api-access-54z9x\") pod \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") "
Jul 02 14:01:46.930683 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.930471    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kserve-provision-location\") pod \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") "
Jul 02 14:01:46.930683 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.930535    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-cabundle-cert\") pod \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") "
Jul 02 14:01:46.930683 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.930582    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-rbac-proxy-sar-config\") pod \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") "
Jul 02 14:01:46.930683 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.930601    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls\") pod \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\" (UID: \"d195ac35-8371-4f55-96a1-f3e8b5c785a5\") "
Jul 02 14:01:46.930917 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.930817    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d195ac35-8371-4f55-96a1-f3e8b5c785a5" (UID: "d195ac35-8371-4f55-96a1-f3e8b5c785a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:01:46.930990 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.930964    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d195ac35-8371-4f55-96a1-f3e8b5c785a5" (UID: "d195ac35-8371-4f55-96a1-f3e8b5c785a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:01:46.931027 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.930965    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d195ac35-8371-4f55-96a1-f3e8b5c785a5" (UID: "d195ac35-8371-4f55-96a1-f3e8b5c785a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:01:46.932662 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.932632    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d195ac35-8371-4f55-96a1-f3e8b5c785a5" (UID: "d195ac35-8371-4f55-96a1-f3e8b5c785a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:01:46.932775 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:46.932687    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-api-access-54z9x" (OuterVolumeSpecName: "kube-api-access-54z9x") pod "d195ac35-8371-4f55-96a1-f3e8b5c785a5" (UID: "d195ac35-8371-4f55-96a1-f3e8b5c785a5"). InnerVolumeSpecName "kube-api-access-54z9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:01:47.032028 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.031943    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:01:47.032028 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.031974    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d195ac35-8371-4f55-96a1-f3e8b5c785a5-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:01:47.032028 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.031985    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54z9x\" (UniqueName: \"kubernetes.io/projected/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kube-api-access-54z9x\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:01:47.032028 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.031995    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d195ac35-8371-4f55-96a1-f3e8b5c785a5-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:01:47.032028 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.032005    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d195ac35-8371-4f55-96a1-f3e8b5c785a5-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:01:47.291700 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.291619    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"
Jul 02 14:01:47.291860 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.291622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n" event={"ID":"d195ac35-8371-4f55-96a1-f3e8b5c785a5","Type":"ContainerDied","Data":"c2f0d14850a820dd503c63a029ad311fb9bf815dc4fc72db1d7975e45c8507f6"}
Jul 02 14:01:47.291860 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.291764    2566 scope.go:117] "RemoveContainer" containerID="94ea480f3c77164b9e886993bc277eb7d1a0de32bdfe1a744ece2962756b4c16"
Jul 02 14:01:47.300862 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.300845    2566 scope.go:117] "RemoveContainer" containerID="0c922f3edcca09b1d4b2f1b85c5babf6b5bf81a77e048888e5efe9fedbdd89b6"
Jul 02 14:01:47.308647 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.308626    2566 scope.go:117] "RemoveContainer" containerID="4b54ce651c541f0508052d2f046f7761e2134540ba7f194f96fb1cfc751ed0ef"
Jul 02 14:01:47.316413 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.316390    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"]
Jul 02 14:01:47.317005 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.316981    2566 scope.go:117] "RemoveContainer" containerID="b434e14902d9499303c41677bc4cc3e89bfd202bfedff71598c8d3d083b26d6b"
Jul 02 14:01:47.320868 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:47.320846    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-1b7e6-predictor-7ff7d59b75-fpd2n"]
Jul 02 14:01:48.503156 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:48.503123    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" path="/var/lib/kubelet/pods/d195ac35-8371-4f55-96a1-f3e8b5c785a5/volumes"
Jul 02 14:01:49.178835 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:49.178801    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:01:59.178822 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:01:59.178778    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:02:01.282501 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:02:01.282448    2566 scope.go:117] "RemoveContainer" containerID="9a27fcb8f64377f5f3a4592970eac016908e69d8aa338d3d4cf498913a04b0ca"
Jul 02 14:02:01.290984 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:02:01.290967    2566 scope.go:117] "RemoveContainer" containerID="f87839755807bba1c475e0265f61a19c75cf9610107d09d67caef585b0e936c6"
Jul 02 14:02:01.298558 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:02:01.298541    2566 scope.go:117] "RemoveContainer" containerID="58181b3315597004148b509eed1f88b32e13adad4ceb808188bae76e26b12b4d"
Jul 02 14:02:09.178549 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:02:09.178509    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:02:19.179446 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:02:19.179405    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:02:29.179418 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:02:29.179383    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:02:39.178731 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:02:39.178691    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:02:49.178527 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:02:49.178465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:02:52.501713 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:02:52.501620    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:03:02.501683 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:03:02.501634    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:03:12.501854 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:03:12.501809    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:03:22.502156 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:03:22.502110    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:03:32.501635 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:03:32.501595    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:03:42.506091 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:03:42.504232    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:03:52.501637 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:03:52.501599    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:04:02.503732 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:02.503698    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:04:06.452272 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:06.452238    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"]
Jul 02 14:04:06.452799 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:06.452527    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" containerID="cri-o://52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b" gracePeriod=30
Jul 02 14:04:06.452799 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:06.452563    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kube-rbac-proxy" containerID="cri-o://bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46" gracePeriod=30
Jul 02 14:04:06.911843 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:06.911761    2566 generic.go:356] "Generic (PLEG): container finished" podID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerID="bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46" exitCode=2
Jul 02 14:04:06.911843 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:06.911833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" event={"ID":"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3","Type":"ContainerDied","Data":"bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46"}
Jul 02 14:04:09.174288 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:09.174239    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 02 14:04:11.611690 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.611652    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"]
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.611995    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerName="kserve-container"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612006    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerName="kserve-container"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612019    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612026    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612036    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerName="kube-rbac-proxy"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612043    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerName="kube-rbac-proxy"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612049    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612054    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612064    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612069    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612081    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="storage-initializer"
Jul 02 14:04:11.612130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612087    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="storage-initializer"
Jul 02 14:04:11.612563 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612145    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="agent"
Jul 02 14:04:11.612563 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612153    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerName="kserve-container"
Jul 02 14:04:11.612563 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612158    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kube-rbac-proxy"
Jul 02 14:04:11.612563 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612164    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d195ac35-8371-4f55-96a1-f3e8b5c785a5" containerName="kserve-container"
Jul 02 14:04:11.612563 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.612170    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0558b3-ff74-4543-bd36-79ba2e659422" containerName="kube-rbac-proxy"
Jul 02 14:04:11.615567 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.615549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.618175 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.618153    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6f2d5c-kube-rbac-proxy-sar-config\""
Jul 02 14:04:11.618175 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.618171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6f2d5c-predictor-serving-cert\""
Jul 02 14:04:11.625605 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.625580    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"]
Jul 02 14:04:11.666401 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.666369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96ed8409-2230-4cd3-94b8-3740dae79b3a-proxy-tls\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.666401 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.666410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-api-access-klpft\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.666644 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.666446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-cabundle-cert\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.666644 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.666568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96ed8409-2230-4cd3-94b8-3740dae79b3a-kserve-provision-location\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.666644 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.666611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.767023 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.766979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-cabundle-cert\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.767221 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.767046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96ed8409-2230-4cd3-94b8-3740dae79b3a-kserve-provision-location\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.767221 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.767070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.767221 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.767095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96ed8409-2230-4cd3-94b8-3740dae79b3a-proxy-tls\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.767221 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.767120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-api-access-klpft\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.767605 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.767583    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96ed8409-2230-4cd3-94b8-3740dae79b3a-kserve-provision-location\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.767845 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.767824    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.767845 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.767838    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-cabundle-cert\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.769711 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.769694    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96ed8409-2230-4cd3-94b8-3740dae79b3a-proxy-tls\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.775887 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.775867    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-api-access-klpft\") pod \"isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") " pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:11.927260 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:11.927162    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:12.067056 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:12.067031    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"]
Jul 02 14:04:12.069446 ip-10-0-136-2 kubenswrapper[2566]: W0702 14:04:12.069418    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ed8409_2230_4cd3_94b8_3740dae79b3a.slice/crio-88aecc59f8adf0dc2b37386a18d1fe988dc116bd84160f5f41db079e46abed5d WatchSource:0}: Error finding container 88aecc59f8adf0dc2b37386a18d1fe988dc116bd84160f5f41db079e46abed5d: Status 404 returned error can't find the container with id 88aecc59f8adf0dc2b37386a18d1fe988dc116bd84160f5f41db079e46abed5d
Jul 02 14:04:12.502162 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:12.502123    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:04:12.943215 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:12.943183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" event={"ID":"96ed8409-2230-4cd3-94b8-3740dae79b3a","Type":"ContainerStarted","Data":"d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0"}
Jul 02 14:04:12.943611 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:12.943242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" event={"ID":"96ed8409-2230-4cd3-94b8-3740dae79b3a","Type":"ContainerStarted","Data":"88aecc59f8adf0dc2b37386a18d1fe988dc116bd84160f5f41db079e46abed5d"}
Jul 02 14:04:14.173771 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:14.173729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 02 14:04:15.959549 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:15.959518    2566 generic.go:356] "Generic (PLEG): container finished" podID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerID="d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0" exitCode=0
Jul 02 14:04:15.959941 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:15.959566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" event={"ID":"96ed8409-2230-4cd3-94b8-3740dae79b3a","Type":"ContainerDied","Data":"d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0"}
Jul 02 14:04:16.499385 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.499363    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:04:16.605805 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.605727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpmbz\" (UniqueName: \"kubernetes.io/projected/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-api-access-lpmbz\") pod \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") "
Jul 02 14:04:16.605805 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.605772    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-proxy-tls\") pod \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") "
Jul 02 14:04:16.605805 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.605801    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-rbac-proxy-sar-config\") pod \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") "
Jul 02 14:04:16.606102 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.605932    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-cabundle-cert\") pod \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") "
Jul 02 14:04:16.606102 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.606052    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kserve-provision-location\") pod \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\" (UID: \"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3\") "
Jul 02 14:04:16.606203 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.606158    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" (UID: "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:04:16.606280 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.606253    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" (UID: "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:04:16.606547 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.606522    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" (UID: "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:04:16.607406 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.606822    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:04:16.607406 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.606848    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:04:16.607406 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.606870    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:04:16.608192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.608172    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-api-access-lpmbz" (OuterVolumeSpecName: "kube-api-access-lpmbz") pod "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" (UID: "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3"). InnerVolumeSpecName "kube-api-access-lpmbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:04:16.608337 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.608318    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" (UID: "0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:04:16.708130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.708090    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpmbz\" (UniqueName: \"kubernetes.io/projected/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-kube-api-access-lpmbz\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:04:16.708130 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.708125    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:04:16.966315 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.966278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" event={"ID":"96ed8409-2230-4cd3-94b8-3740dae79b3a","Type":"ContainerStarted","Data":"56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6"}
Jul 02 14:04:16.966768 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.966348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" event={"ID":"96ed8409-2230-4cd3-94b8-3740dae79b3a","Type":"ContainerStarted","Data":"4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22"}
Jul 02 14:04:16.966768 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.966624    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:16.968256 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.968235    2566 generic.go:356] "Generic (PLEG): container finished" podID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerID="52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b" exitCode=0
Jul 02 14:04:16.968366 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.968287    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"
Jul 02 14:04:16.968366 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.968312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" event={"ID":"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3","Type":"ContainerDied","Data":"52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b"}
Jul 02 14:04:16.968431 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.968371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2" event={"ID":"0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3","Type":"ContainerDied","Data":"295affae33f84cdeb465b1e3590e4ea867ed96a58e5a9d40756e6c7c7fc67936"}
Jul 02 14:04:16.968431 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.968411    2566 scope.go:117] "RemoveContainer" containerID="bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46"
Jul 02 14:04:16.977416 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.977397    2566 scope.go:117] "RemoveContainer" containerID="52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b"
Jul 02 14:04:16.987981 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.987949    2566 scope.go:117] "RemoveContainer" containerID="9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a"
Jul 02 14:04:16.989217 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.989177    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podStartSLOduration=2.523962375 podStartE2EDuration="5.989162436s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.465200061s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:04:16.987094779 +0000 UTC m=+1156.828732293" watchObservedRunningTime="2026-07-02 14:04:16.989162436 +0000 UTC m=+1156.830799950"
Jul 02 14:04:16.996159 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.996142    2566 scope.go:117] "RemoveContainer" containerID="bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46"
Jul 02 14:04:16.996409 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:04:16.996392    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46\": container with ID starting with bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46 not found: ID does not exist" containerID="bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46"
Jul 02 14:04:16.996460 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.996416    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46"} err="failed to get container status \"bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46\": rpc error: code = NotFound desc = could not find container \"bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46\": container with ID starting with bad905e70d62422662dc94886cfeef97997061c369b1ed70d5e1ae5468456f46 not found: ID does not exist"
Jul 02 14:04:16.996460 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.996432    2566 scope.go:117] "RemoveContainer" containerID="52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b"
Jul 02 14:04:16.996679 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:04:16.996661    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b\": container with ID starting with 52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b not found: ID does not exist" containerID="52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b"
Jul 02 14:04:16.996727 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.996685    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b"} err="failed to get container status \"52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b\": rpc error: code = NotFound desc = could not find container \"52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b\": container with ID starting with 52c1fae3bb3d8c6f539bf6da5f907fe25f4a6e7845e869693469e6326322845b not found: ID does not exist"
Jul 02 14:04:16.996727 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.996702    2566 scope.go:117] "RemoveContainer" containerID="9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a"
Jul 02 14:04:16.996940 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:04:16.996922    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a\": container with ID starting with 9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a not found: ID does not exist" containerID="9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a"
Jul 02 14:04:16.996994 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:16.996945    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a"} err="failed to get container status \"9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a\": rpc error: code = NotFound desc = could not find container \"9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a\": container with ID starting with 9de6683440949371546b7773d0e72f5808c30680369f8f9d40bf0f7611141c3a not found: ID does not exist"
Jul 02 14:04:17.003384 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:17.003358    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"]
Jul 02 14:04:17.005188 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:17.005170    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1c082-predictor-9f4ff4b7-z4js2"]
Jul 02 14:04:17.980016 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:17.979986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:17.981445 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:17.981414    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:04:18.503738 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:18.503706    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" path="/var/lib/kubelet/pods/0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3/volumes"
Jul 02 14:04:18.984319 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:18.984279    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:04:23.989409 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:23.989336    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:04:23.989933 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:23.989906    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:04:33.990426 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:33.990385    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:04:43.990772 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:43.990730    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:04:53.990637 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:04:53.990587    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:05:01.236356 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:01.236325    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 14:05:01.238220 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:01.238197    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 14:05:03.990739 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:03.990700    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:05:13.990207 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:13.990166    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:05:23.990813 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:23.990767    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:05:33.990686 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:33.990653    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:05:41.836706 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.836666    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"]
Jul 02 14:05:41.837192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.837019    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kube-rbac-proxy"
Jul 02 14:05:41.837192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.837031    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kube-rbac-proxy"
Jul 02 14:05:41.837192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.837048    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container"
Jul 02 14:05:41.837192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.837053    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container"
Jul 02 14:05:41.837192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.837062    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="storage-initializer"
Jul 02 14:05:41.837192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.837068    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="storage-initializer"
Jul 02 14:05:41.837192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.837129    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kserve-container"
Jul 02 14:05:41.837192 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.837142    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc540ec-5678-4ea3-8a6b-5f8ffc43ecd3" containerName="kube-rbac-proxy"
Jul 02 14:05:41.840288 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.840269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:41.842976 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.842942    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6f2d5c-kube-rbac-proxy-sar-config\""
Jul 02 14:05:41.843220 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.843008    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6f2d5c-predictor-serving-cert\""
Jul 02 14:05:41.843358 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.843017    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-6f2d5c\""
Jul 02 14:05:41.843358 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.843113    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-6f2d5c-dockercfg-wbz2q\""
Jul 02 14:05:41.843473 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.843427    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"]
Jul 02 14:05:41.935976 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.935934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:41.935976 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.935980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074459bf-3492-4437-ab49-09a6381701cf-kserve-provision-location\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:41.936209 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.936006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9lt\" (UniqueName: \"kubernetes.io/projected/074459bf-3492-4437-ab49-09a6381701cf-kube-api-access-pm9lt\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:41.936209 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.936084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-cabundle-cert\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:41.936209 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:41.936153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074459bf-3492-4437-ab49-09a6381701cf-proxy-tls\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.036647 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.036600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-cabundle-cert\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.036855 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.036686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074459bf-3492-4437-ab49-09a6381701cf-proxy-tls\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.036855 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.036712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.036855 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.036734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074459bf-3492-4437-ab49-09a6381701cf-kserve-provision-location\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.036855 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.036762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9lt\" (UniqueName: \"kubernetes.io/projected/074459bf-3492-4437-ab49-09a6381701cf-kube-api-access-pm9lt\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.037179 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.037156    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074459bf-3492-4437-ab49-09a6381701cf-kserve-provision-location\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.037358 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.037335    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.037421 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.037358    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-cabundle-cert\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.039148 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.039129    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074459bf-3492-4437-ab49-09a6381701cf-proxy-tls\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.046967 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.046943    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9lt\" (UniqueName: \"kubernetes.io/projected/074459bf-3492-4437-ab49-09a6381701cf-kube-api-access-pm9lt\") pod \"isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") " pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.152398 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.152294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:05:42.290372 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.290344    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"]
Jul 02 14:05:42.293085 ip-10-0-136-2 kubenswrapper[2566]: W0702 14:05:42.293058    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074459bf_3492_4437_ab49_09a6381701cf.slice/crio-1f9d0cb58cebb45b8ea40f2d86a57b3755a75fa289d824affb8920fcdca9d536 WatchSource:0}: Error finding container 1f9d0cb58cebb45b8ea40f2d86a57b3755a75fa289d824affb8920fcdca9d536: Status 404 returned error can't find the container with id 1f9d0cb58cebb45b8ea40f2d86a57b3755a75fa289d824affb8920fcdca9d536
Jul 02 14:05:42.342345 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:42.342311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj" event={"ID":"074459bf-3492-4437-ab49-09a6381701cf","Type":"ContainerStarted","Data":"1f9d0cb58cebb45b8ea40f2d86a57b3755a75fa289d824affb8920fcdca9d536"}
Jul 02 14:05:43.347768 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:43.347733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj" event={"ID":"074459bf-3492-4437-ab49-09a6381701cf","Type":"ContainerStarted","Data":"65a1c7918b3f6ee4174a35acc6483e471c36ca68e7d54dbb20a9edd4249ffd37"}
Jul 02 14:05:48.372433 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:48.372403    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_074459bf-3492-4437-ab49-09a6381701cf/storage-initializer/0.log"
Jul 02 14:05:48.372851 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:48.372443    2566 generic.go:356] "Generic (PLEG): container finished" podID="074459bf-3492-4437-ab49-09a6381701cf" containerID="65a1c7918b3f6ee4174a35acc6483e471c36ca68e7d54dbb20a9edd4249ffd37" exitCode=1
Jul 02 14:05:48.372851 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:48.372509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj" event={"ID":"074459bf-3492-4437-ab49-09a6381701cf","Type":"ContainerDied","Data":"65a1c7918b3f6ee4174a35acc6483e471c36ca68e7d54dbb20a9edd4249ffd37"}
Jul 02 14:05:49.379049 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:49.379020    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_074459bf-3492-4437-ab49-09a6381701cf/storage-initializer/0.log"
Jul 02 14:05:49.379429 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:49.379084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj" event={"ID":"074459bf-3492-4437-ab49-09a6381701cf","Type":"ContainerStarted","Data":"70b05f28285b3b68edb4177aa374937938493c6a935c6216e38ee43345e027eb"}
Jul 02 14:05:55.403971 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:55.403942    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_074459bf-3492-4437-ab49-09a6381701cf/storage-initializer/1.log"
Jul 02 14:05:55.404431 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:55.404288    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_074459bf-3492-4437-ab49-09a6381701cf/storage-initializer/0.log"
Jul 02 14:05:55.404431 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:55.404317    2566 generic.go:356] "Generic (PLEG): container finished" podID="074459bf-3492-4437-ab49-09a6381701cf" containerID="70b05f28285b3b68edb4177aa374937938493c6a935c6216e38ee43345e027eb" exitCode=1
Jul 02 14:05:55.404431 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:55.404391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj" event={"ID":"074459bf-3492-4437-ab49-09a6381701cf","Type":"ContainerDied","Data":"70b05f28285b3b68edb4177aa374937938493c6a935c6216e38ee43345e027eb"}
Jul 02 14:05:55.404565 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:55.404452    2566 scope.go:117] "RemoveContainer" containerID="65a1c7918b3f6ee4174a35acc6483e471c36ca68e7d54dbb20a9edd4249ffd37"
Jul 02 14:05:55.404799 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:55.404776    2566 scope.go:117] "RemoveContainer" containerID="65a1c7918b3f6ee4174a35acc6483e471c36ca68e7d54dbb20a9edd4249ffd37"
Jul 02 14:05:55.415743 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:05:55.415719    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_kserve-ci-e2e-test(074459bf-3492-4437-ab49-09a6381701cf)\"" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj" podUID="074459bf-3492-4437-ab49-09a6381701cf"
Jul 02 14:05:55.415857 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:05:55.415833    2566 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_kserve-ci-e2e-test_074459bf-3492-4437-ab49-09a6381701cf_0 in pod sandbox 1f9d0cb58cebb45b8ea40f2d86a57b3755a75fa289d824affb8920fcdca9d536 from index: no such id: '65a1c7918b3f6ee4174a35acc6483e471c36ca68e7d54dbb20a9edd4249ffd37'" containerID="65a1c7918b3f6ee4174a35acc6483e471c36ca68e7d54dbb20a9edd4249ffd37"
Jul 02 14:05:55.415910 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:55.415866    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a1c7918b3f6ee4174a35acc6483e471c36ca68e7d54dbb20a9edd4249ffd37"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_kserve-ci-e2e-test_074459bf-3492-4437-ab49-09a6381701cf_0 in pod sandbox 1f9d0cb58cebb45b8ea40f2d86a57b3755a75fa289d824affb8920fcdca9d536 from index: no such id: '65a1c7918b3f6ee4174a35acc6483e471c36ca68e7d54dbb20a9edd4249ffd37'"
Jul 02 14:05:56.409784 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:56.409755    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_074459bf-3492-4437-ab49-09a6381701cf/storage-initializer/1.log"
Jul 02 14:05:59.868390 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:59.868354    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"]
Jul 02 14:05:59.931032 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:59.931006    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"]
Jul 02 14:05:59.931318 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:59.931263    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" containerID="cri-o://4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22" gracePeriod=30
Jul 02 14:05:59.931524 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:05:59.931310    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kube-rbac-proxy" containerID="cri-o://56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6" gracePeriod=30
Jul 02 14:06:00.053061 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.053041    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_074459bf-3492-4437-ab49-09a6381701cf/storage-initializer/1.log"
Jul 02 14:06:00.054948 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.054934    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:06:00.087728 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.087702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm9lt\" (UniqueName: \"kubernetes.io/projected/074459bf-3492-4437-ab49-09a6381701cf-kube-api-access-pm9lt\") pod \"074459bf-3492-4437-ab49-09a6381701cf\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") "
Jul 02 14:06:00.087871 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.087780    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-cabundle-cert\") pod \"074459bf-3492-4437-ab49-09a6381701cf\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") "
Jul 02 14:06:00.087871 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.087835    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074459bf-3492-4437-ab49-09a6381701cf-kserve-provision-location\") pod \"074459bf-3492-4437-ab49-09a6381701cf\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") "
Jul 02 14:06:00.087969 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.087908    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074459bf-3492-4437-ab49-09a6381701cf-proxy-tls\") pod \"074459bf-3492-4437-ab49-09a6381701cf\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") "
Jul 02 14:06:00.087969 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.087948    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-kube-rbac-proxy-sar-config\") pod \"074459bf-3492-4437-ab49-09a6381701cf\" (UID: \"074459bf-3492-4437-ab49-09a6381701cf\") "
Jul 02 14:06:00.088179 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.088148    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "074459bf-3492-4437-ab49-09a6381701cf" (UID: "074459bf-3492-4437-ab49-09a6381701cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:06:00.088293 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.088150    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074459bf-3492-4437-ab49-09a6381701cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "074459bf-3492-4437-ab49-09a6381701cf" (UID: "074459bf-3492-4437-ab49-09a6381701cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:06:00.088384 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.088364    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "074459bf-3492-4437-ab49-09a6381701cf" (UID: "074459bf-3492-4437-ab49-09a6381701cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:06:00.089865 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.089839    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074459bf-3492-4437-ab49-09a6381701cf-kube-api-access-pm9lt" (OuterVolumeSpecName: "kube-api-access-pm9lt") pod "074459bf-3492-4437-ab49-09a6381701cf" (UID: "074459bf-3492-4437-ab49-09a6381701cf"). InnerVolumeSpecName "kube-api-access-pm9lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:06:00.089950 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.089911    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074459bf-3492-4437-ab49-09a6381701cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "074459bf-3492-4437-ab49-09a6381701cf" (UID: "074459bf-3492-4437-ab49-09a6381701cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:06:00.189210 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.189181    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9lt\" (UniqueName: \"kubernetes.io/projected/074459bf-3492-4437-ab49-09a6381701cf-kube-api-access-pm9lt\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:00.189210 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.189210    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:00.189390 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.189220    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074459bf-3492-4437-ab49-09a6381701cf-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:00.189390 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.189229    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074459bf-3492-4437-ab49-09a6381701cf-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:00.189390 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.189238    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074459bf-3492-4437-ab49-09a6381701cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:00.429582 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.429551    2566 generic.go:356] "Generic (PLEG): container finished" podID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerID="56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6" exitCode=2
Jul 02 14:06:00.429764 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.429628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" event={"ID":"96ed8409-2230-4cd3-94b8-3740dae79b3a","Type":"ContainerDied","Data":"56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6"}
Jul 02 14:06:00.430746 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.430730    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj_074459bf-3492-4437-ab49-09a6381701cf/storage-initializer/1.log"
Jul 02 14:06:00.430854 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.430813    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"
Jul 02 14:06:00.430854 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.430818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj" event={"ID":"074459bf-3492-4437-ab49-09a6381701cf","Type":"ContainerDied","Data":"1f9d0cb58cebb45b8ea40f2d86a57b3755a75fa289d824affb8920fcdca9d536"}
Jul 02 14:06:00.430917 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.430865    2566 scope.go:117] "RemoveContainer" containerID="70b05f28285b3b68edb4177aa374937938493c6a935c6216e38ee43345e027eb"
Jul 02 14:06:00.467164 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.467136    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"]
Jul 02 14:06:00.470870 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.470848    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6f2d5c-predictor-665b6cbc47-6jrkj"]
Jul 02 14:06:00.503650 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:00.503629    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="074459bf-3492-4437-ab49-09a6381701cf" path="/var/lib/kubelet/pods/074459bf-3492-4437-ab49-09a6381701cf/volumes"
Jul 02 14:06:03.985237 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:03.985203    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 02 14:06:03.990605 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:03.990578    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:06:04.284233 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.284202    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:06:04.325361 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.325336    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96ed8409-2230-4cd3-94b8-3740dae79b3a-proxy-tls\") pod \"96ed8409-2230-4cd3-94b8-3740dae79b3a\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") "
Jul 02 14:06:04.325473 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.325367    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-cabundle-cert\") pod \"96ed8409-2230-4cd3-94b8-3740dae79b3a\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") "
Jul 02 14:06:04.325473 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.325420    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-rbac-proxy-sar-config\") pod \"96ed8409-2230-4cd3-94b8-3740dae79b3a\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") "
Jul 02 14:06:04.325473 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.325444    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96ed8409-2230-4cd3-94b8-3740dae79b3a-kserve-provision-location\") pod \"96ed8409-2230-4cd3-94b8-3740dae79b3a\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") "
Jul 02 14:06:04.325639 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.325478    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-api-access-klpft\") pod \"96ed8409-2230-4cd3-94b8-3740dae79b3a\" (UID: \"96ed8409-2230-4cd3-94b8-3740dae79b3a\") "
Jul 02 14:06:04.325804 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.325771    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96ed8409-2230-4cd3-94b8-3740dae79b3a" (UID: "96ed8409-2230-4cd3-94b8-3740dae79b3a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:06:04.325911 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.325802    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96ed8409-2230-4cd3-94b8-3740dae79b3a" (UID: "96ed8409-2230-4cd3-94b8-3740dae79b3a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:06:04.325911 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.325802    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ed8409-2230-4cd3-94b8-3740dae79b3a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96ed8409-2230-4cd3-94b8-3740dae79b3a" (UID: "96ed8409-2230-4cd3-94b8-3740dae79b3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:06:04.327521 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.327497    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ed8409-2230-4cd3-94b8-3740dae79b3a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96ed8409-2230-4cd3-94b8-3740dae79b3a" (UID: "96ed8409-2230-4cd3-94b8-3740dae79b3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:06:04.327644 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.327620    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-api-access-klpft" (OuterVolumeSpecName: "kube-api-access-klpft") pod "96ed8409-2230-4cd3-94b8-3740dae79b3a" (UID: "96ed8409-2230-4cd3-94b8-3740dae79b3a"). InnerVolumeSpecName "kube-api-access-klpft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:06:04.426691 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.426671    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klpft\" (UniqueName: \"kubernetes.io/projected/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-api-access-klpft\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:04.426691 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.426690    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96ed8409-2230-4cd3-94b8-3740dae79b3a-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:04.426822 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.426699    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:04.426822 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.426708    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96ed8409-2230-4cd3-94b8-3740dae79b3a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:04.426822 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.426716    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96ed8409-2230-4cd3-94b8-3740dae79b3a-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:04.449156 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.449127    2566 generic.go:356] "Generic (PLEG): container finished" podID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerID="4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22" exitCode=0
Jul 02 14:06:04.449242 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.449182    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"
Jul 02 14:06:04.449242 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.449204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" event={"ID":"96ed8409-2230-4cd3-94b8-3740dae79b3a","Type":"ContainerDied","Data":"4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22"}
Jul 02 14:06:04.449320 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.449288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58" event={"ID":"96ed8409-2230-4cd3-94b8-3740dae79b3a","Type":"ContainerDied","Data":"88aecc59f8adf0dc2b37386a18d1fe988dc116bd84160f5f41db079e46abed5d"}
Jul 02 14:06:04.449320 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.449315    2566 scope.go:117] "RemoveContainer" containerID="56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6"
Jul 02 14:06:04.457942 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.457927    2566 scope.go:117] "RemoveContainer" containerID="4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22"
Jul 02 14:06:04.466678 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.466662    2566 scope.go:117] "RemoveContainer" containerID="d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0"
Jul 02 14:06:04.472355 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.472333    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"]
Jul 02 14:06:04.473864 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.473850    2566 scope.go:117] "RemoveContainer" containerID="56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6"
Jul 02 14:06:04.474088 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:06:04.474071    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6\": container with ID starting with 56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6 not found: ID does not exist" containerID="56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6"
Jul 02 14:06:04.474139 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.474094    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6"} err="failed to get container status \"56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6\": rpc error: code = NotFound desc = could not find container \"56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6\": container with ID starting with 56e496d0fbf9ac93f3bdacf3c416f9a3d3fce3e3210424b1c3c549d21a03f2b6 not found: ID does not exist"
Jul 02 14:06:04.474139 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.474111    2566 scope.go:117] "RemoveContainer" containerID="4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22"
Jul 02 14:06:04.474358 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:06:04.474337    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22\": container with ID starting with 4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22 not found: ID does not exist" containerID="4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22"
Jul 02 14:06:04.474409 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.474366    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22"} err="failed to get container status \"4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22\": rpc error: code = NotFound desc = could not find container \"4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22\": container with ID starting with 4e17d69c093584ff153b8e05a99ed296813d8501eb27821e4a2d168f5b416a22 not found: ID does not exist"
Jul 02 14:06:04.474409 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.474389    2566 scope.go:117] "RemoveContainer" containerID="d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0"
Jul 02 14:06:04.474662 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:06:04.474642    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0\": container with ID starting with d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0 not found: ID does not exist" containerID="d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0"
Jul 02 14:06:04.474759 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.474664    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0"} err="failed to get container status \"d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0\": rpc error: code = NotFound desc = could not find container \"d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0\": container with ID starting with d6b42d4fba9a723012bcde6dca1eca314acbc045de1f9c03b7491a7805c909d0 not found: ID does not exist"
Jul 02 14:06:04.476957 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.476938    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6f2d5c-predictor-8588fb45dd-wcq58"]
Jul 02 14:06:04.503085 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:04.503063    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" path="/var/lib/kubelet/pods/96ed8409-2230-4cd3-94b8-3740dae79b3a/volumes"
Jul 02 14:06:05.054748 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.054711    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"]
Jul 02 14:06:05.055218 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055198    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kube-rbac-proxy"
Jul 02 14:06:05.055311 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055222    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kube-rbac-proxy"
Jul 02 14:06:05.055311 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055240    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container"
Jul 02 14:06:05.055311 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055249    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container"
Jul 02 14:06:05.055311 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055259    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074459bf-3492-4437-ab49-09a6381701cf" containerName="storage-initializer"
Jul 02 14:06:05.055311 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055267    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="074459bf-3492-4437-ab49-09a6381701cf" containerName="storage-initializer"
Jul 02 14:06:05.055311 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055288    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074459bf-3492-4437-ab49-09a6381701cf" containerName="storage-initializer"
Jul 02 14:06:05.055311 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055297    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="074459bf-3492-4437-ab49-09a6381701cf" containerName="storage-initializer"
Jul 02 14:06:05.055311 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055310    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="storage-initializer"
Jul 02 14:06:05.055728 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055319    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="storage-initializer"
Jul 02 14:06:05.055728 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055400    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kserve-container"
Jul 02 14:06:05.055728 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055414    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="074459bf-3492-4437-ab49-09a6381701cf" containerName="storage-initializer"
Jul 02 14:06:05.055728 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055427    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="074459bf-3492-4437-ab49-09a6381701cf" containerName="storage-initializer"
Jul 02 14:06:05.055728 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.055438    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ed8409-2230-4cd3-94b8-3740dae79b3a" containerName="kube-rbac-proxy"
Jul 02 14:06:05.060899 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.060879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.063574 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.063554    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2d1354-predictor-serving-cert\""
Jul 02 14:06:05.063696 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.063596    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 14:06:05.063696 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.063606    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-2d1354\""
Jul 02 14:06:05.063696 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.063628    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2d1354-kube-rbac-proxy-sar-config\""
Jul 02 14:06:05.063872 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.063737    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 14:06:05.063991 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.063974    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:06:05.064382 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.064367    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-2d1354-dockercfg-248qj\""
Jul 02 14:06:05.068824 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.068794    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"]
Jul 02 14:06:05.132013 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.131993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kserve-provision-location\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.132132 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.132042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xxk\" (UniqueName: \"kubernetes.io/projected/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-api-access-h9xxk\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.132132 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.132069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.132132 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.132108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-proxy-tls\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.132249 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.132141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-cabundle-cert\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.233433 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.233408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kserve-provision-location\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.233564 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.233461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xxk\" (UniqueName: \"kubernetes.io/projected/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-api-access-h9xxk\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.233564 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.233498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.233564 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.233518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-proxy-tls\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.233564 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.233549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-cabundle-cert\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.233850 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.233828    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kserve-provision-location\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.234105 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.234084    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.234199 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.234155    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-cabundle-cert\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.235805 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.235789    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-proxy-tls\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.242110 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.242089    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xxk\" (UniqueName: \"kubernetes.io/projected/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-api-access-h9xxk\") pod \"isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") " pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.373319 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.373268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:05.502955 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:05.502928    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"]
Jul 02 14:06:05.504861 ip-10-0-136-2 kubenswrapper[2566]: W0702 14:06:05.504833    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3d4aa0_bd19_4b1e_8066_15eb44dfd87b.slice/crio-9cd7262827e60755a54760c2a3862fbaadd3dda5ad3fd92095a663bfed161c75 WatchSource:0}: Error finding container 9cd7262827e60755a54760c2a3862fbaadd3dda5ad3fd92095a663bfed161c75: Status 404 returned error can't find the container with id 9cd7262827e60755a54760c2a3862fbaadd3dda5ad3fd92095a663bfed161c75
Jul 02 14:06:06.460419 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:06.460385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8" event={"ID":"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b","Type":"ContainerStarted","Data":"9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d"}
Jul 02 14:06:06.460867 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:06.460467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8" event={"ID":"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b","Type":"ContainerStarted","Data":"9cd7262827e60755a54760c2a3862fbaadd3dda5ad3fd92095a663bfed161c75"}
Jul 02 14:06:10.477793 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:10.477723    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8_ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b/storage-initializer/0.log"
Jul 02 14:06:10.477793 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:10.477758    2566 generic.go:356] "Generic (PLEG): container finished" podID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" containerID="9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d" exitCode=1
Jul 02 14:06:10.478179 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:10.477803    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8" event={"ID":"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b","Type":"ContainerDied","Data":"9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d"}
Jul 02 14:06:11.489434 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:11.489405    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8_ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b/storage-initializer/0.log"
Jul 02 14:06:11.489880 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:11.489457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8" event={"ID":"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b","Type":"ContainerStarted","Data":"2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db"}
Jul 02 14:06:15.048689 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:15.048657    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"]
Jul 02 14:06:15.049100 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:15.048858    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8" podUID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" containerName="storage-initializer" containerID="cri-o://2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db" gracePeriod=30
Jul 02 14:06:16.191236 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.191216    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8_ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b/storage-initializer/1.log"
Jul 02 14:06:16.191615 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.191601    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8_ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b/storage-initializer/0.log"
Jul 02 14:06:16.193901 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.193887    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:16.331011 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.330940    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-proxy-tls\") pod \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") "
Jul 02 14:06:16.331011 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.330976    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kserve-provision-location\") pod \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") "
Jul 02 14:06:16.331011 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.330996    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-cabundle-cert\") pod \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") "
Jul 02 14:06:16.331181 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.331060    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-rbac-proxy-sar-config\") pod \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") "
Jul 02 14:06:16.331181 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.331090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9xxk\" (UniqueName: \"kubernetes.io/projected/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-api-access-h9xxk\") pod \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\" (UID: \"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b\") "
Jul 02 14:06:16.331262 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.331239    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" (UID: "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:06:16.331446 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.331417    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" (UID: "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:06:16.331577 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.331421    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" (UID: "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:06:16.332992 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.332970    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" (UID: "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:06:16.333056 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.333027    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-api-access-h9xxk" (OuterVolumeSpecName: "kube-api-access-h9xxk") pod "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" (UID: "ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b"). InnerVolumeSpecName "kube-api-access-h9xxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:06:16.432115 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.432091    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:16.432115 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.432112    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9xxk\" (UniqueName: \"kubernetes.io/projected/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kube-api-access-h9xxk\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:16.432266 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.432122    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:16.432266 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.432131    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:16.432266 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.432141    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:06:16.513007 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.512983    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8_ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b/storage-initializer/1.log"
Jul 02 14:06:16.513390 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.513371    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8_ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b/storage-initializer/0.log"
Jul 02 14:06:16.513464 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.513417    2566 generic.go:356] "Generic (PLEG): container finished" podID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" containerID="2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db" exitCode=1
Jul 02 14:06:16.513608 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.513472    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"
Jul 02 14:06:16.513608 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.513508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8" event={"ID":"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b","Type":"ContainerDied","Data":"2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db"}
Jul 02 14:06:16.513608 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.513580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8" event={"ID":"ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b","Type":"ContainerDied","Data":"9cd7262827e60755a54760c2a3862fbaadd3dda5ad3fd92095a663bfed161c75"}
Jul 02 14:06:16.513764 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.513620    2566 scope.go:117] "RemoveContainer" containerID="2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db"
Jul 02 14:06:16.523446 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.523432    2566 scope.go:117] "RemoveContainer" containerID="9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d"
Jul 02 14:06:16.531063 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.531050    2566 scope.go:117] "RemoveContainer" containerID="2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db"
Jul 02 14:06:16.531279 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:06:16.531264    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db\": container with ID starting with 2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db not found: ID does not exist" containerID="2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db"
Jul 02 14:06:16.531328 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.531285    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db"} err="failed to get container status \"2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db\": rpc error: code = NotFound desc = could not find container \"2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db\": container with ID starting with 2f23e465b362a69d171559dc8e1ed6199612d9d22e6ce5c09a26698da19849db not found: ID does not exist"
Jul 02 14:06:16.531328 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.531301    2566 scope.go:117] "RemoveContainer" containerID="9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d"
Jul 02 14:06:16.531543 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:06:16.531524    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d\": container with ID starting with 9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d not found: ID does not exist" containerID="9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d"
Jul 02 14:06:16.531610 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.531550    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d"} err="failed to get container status \"9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d\": rpc error: code = NotFound desc = could not find container \"9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d\": container with ID starting with 9e36c1d6d520616ff5e8d410e061660def064e7f644d38ae377c7792636d782d not found: ID does not exist"
Jul 02 14:06:16.547946 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.547918    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"]
Jul 02 14:06:16.554390 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:16.554366    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2d1354-predictor-56dcf55695-g5tv8"]
Jul 02 14:06:18.503995 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:18.503954    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" path="/var/lib/kubelet/pods/ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b/volumes"
Jul 02 14:06:20.270189 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.270154    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"]
Jul 02 14:06:20.270572 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.270503    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" containerName="storage-initializer"
Jul 02 14:06:20.270572 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.270515    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" containerName="storage-initializer"
Jul 02 14:06:20.270572 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.270535    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" containerName="storage-initializer"
Jul 02 14:06:20.270572 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.270540    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" containerName="storage-initializer"
Jul 02 14:06:20.270705 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.270592    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" containerName="storage-initializer"
Jul 02 14:06:20.270705 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.270600    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3d4aa0-bd19-4b1e-8066-15eb44dfd87b" containerName="storage-initializer"
Jul 02 14:06:20.275418 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.275397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.277982 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.277962    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-bxgtw\""
Jul 02 14:06:20.277982 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.277973    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:06:20.278243 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.278228    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-e66b7-predictor-serving-cert\""
Jul 02 14:06:20.278330 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.278227    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-e66b7-kube-rbac-proxy-sar-config\""
Jul 02 14:06:20.279043 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.279022    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 14:06:20.279152 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.279046    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:06:20.279152 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.279094    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 14:06:20.283819 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.283799    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"]
Jul 02 14:06:20.366375 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.366344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/156661a7-3632-4237-ab09-f28babdc1b04-proxy-tls\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.366375 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.366380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvv8j\" (UniqueName: \"kubernetes.io/projected/156661a7-3632-4237-ab09-f28babdc1b04-kube-api-access-hvv8j\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.366640 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.366421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.366640 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.366438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/156661a7-3632-4237-ab09-f28babdc1b04-kserve-provision-location\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.366640 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.366545    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-cabundle-cert\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.467608 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.467569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/156661a7-3632-4237-ab09-f28babdc1b04-proxy-tls\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.467608 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.467610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvv8j\" (UniqueName: \"kubernetes.io/projected/156661a7-3632-4237-ab09-f28babdc1b04-kube-api-access-hvv8j\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.467884 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.467646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.467884 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.467662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/156661a7-3632-4237-ab09-f28babdc1b04-kserve-provision-location\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.467884 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.467690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-cabundle-cert\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.468159 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.468129    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/156661a7-3632-4237-ab09-f28babdc1b04-kserve-provision-location\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.468309 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.468289    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-cabundle-cert\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.468380 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.468310    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.470054 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.470033    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/156661a7-3632-4237-ab09-f28babdc1b04-proxy-tls\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.476372 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.476350    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvv8j\" (UniqueName: \"kubernetes.io/projected/156661a7-3632-4237-ab09-f28babdc1b04-kube-api-access-hvv8j\") pod \"raw-sklearn-e66b7-predictor-769988c997-zrpvc\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") " pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.587276 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.587201    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:20.717749 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:20.717719    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"]
Jul 02 14:06:20.720986 ip-10-0-136-2 kubenswrapper[2566]: W0702 14:06:20.720961    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156661a7_3632_4237_ab09_f28babdc1b04.slice/crio-f8f59c461558f14d25ac57c8b10da8bb7d1391923164c8ab1890a1821c5fc624 WatchSource:0}: Error finding container f8f59c461558f14d25ac57c8b10da8bb7d1391923164c8ab1890a1821c5fc624: Status 404 returned error can't find the container with id f8f59c461558f14d25ac57c8b10da8bb7d1391923164c8ab1890a1821c5fc624
Jul 02 14:06:21.543338 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:21.543302    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" event={"ID":"156661a7-3632-4237-ab09-f28babdc1b04","Type":"ContainerStarted","Data":"d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3"}
Jul 02 14:06:21.543719 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:21.543388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" event={"ID":"156661a7-3632-4237-ab09-f28babdc1b04","Type":"ContainerStarted","Data":"f8f59c461558f14d25ac57c8b10da8bb7d1391923164c8ab1890a1821c5fc624"}
Jul 02 14:06:25.561383 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:25.561344    2566 generic.go:356] "Generic (PLEG): container finished" podID="156661a7-3632-4237-ab09-f28babdc1b04" containerID="d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3" exitCode=0
Jul 02 14:06:25.561791 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:25.561409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" event={"ID":"156661a7-3632-4237-ab09-f28babdc1b04","Type":"ContainerDied","Data":"d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3"}
Jul 02 14:06:26.567880 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:26.567846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" event={"ID":"156661a7-3632-4237-ab09-f28babdc1b04","Type":"ContainerStarted","Data":"2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c"}
Jul 02 14:06:26.568374 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:26.567916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" event={"ID":"156661a7-3632-4237-ab09-f28babdc1b04","Type":"ContainerStarted","Data":"dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42"}
Jul 02 14:06:26.568374 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:26.568133    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:26.588731 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:26.588689    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podStartSLOduration=2.788155749 podStartE2EDuration="6.588673517s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800517768s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:06:26.587966648 +0000 UTC m=+1286.429604162" watchObservedRunningTime="2026-07-02 14:06:26.588673517 +0000 UTC m=+1286.430311033"
Jul 02 14:06:27.573131 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:27.573100    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:27.574346 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:27.574313    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:06:28.576997 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:28.576951    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:06:33.581682 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:33.581656    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:06:33.582152 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:33.582128    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:06:43.582357 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:43.582317    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:06:53.582319 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:06:53.582279    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:07:03.582369 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:03.582332    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:07:13.583066 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:13.583027    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:07:23.582775 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:23.582688    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:07:33.582987 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:33.582951    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:07:43.582637 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:43.582601    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:07:50.430942 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:50.430912    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"]
Jul 02 14:07:50.431360 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:50.431130    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" containerID="cri-o://dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42" gracePeriod=30
Jul 02 14:07:50.431360 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:50.431190    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kube-rbac-proxy" containerID="cri-o://2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c" gracePeriod=30
Jul 02 14:07:50.937229 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:50.937201    2566 generic.go:356] "Generic (PLEG): container finished" podID="156661a7-3632-4237-ab09-f28babdc1b04" containerID="2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c" exitCode=2
Jul 02 14:07:50.937374 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:50.937237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" event={"ID":"156661a7-3632-4237-ab09-f28babdc1b04","Type":"ContainerDied","Data":"2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c"}
Jul 02 14:07:53.577671 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:53.577630    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 02 14:07:53.583015 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:53.582990    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:07:54.277349 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.277327    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:07:54.386594 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.386522    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-kube-rbac-proxy-sar-config\") pod \"156661a7-3632-4237-ab09-f28babdc1b04\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") "
Jul 02 14:07:54.386594 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.386577    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvv8j\" (UniqueName: \"kubernetes.io/projected/156661a7-3632-4237-ab09-f28babdc1b04-kube-api-access-hvv8j\") pod \"156661a7-3632-4237-ab09-f28babdc1b04\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") "
Jul 02 14:07:54.386769 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.386609    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/156661a7-3632-4237-ab09-f28babdc1b04-kserve-provision-location\") pod \"156661a7-3632-4237-ab09-f28babdc1b04\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") "
Jul 02 14:07:54.386852 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.386829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-cabundle-cert\") pod \"156661a7-3632-4237-ab09-f28babdc1b04\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") "
Jul 02 14:07:54.386914 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.386812    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "156661a7-3632-4237-ab09-f28babdc1b04" (UID: "156661a7-3632-4237-ab09-f28babdc1b04"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:07:54.386969 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.386920    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/156661a7-3632-4237-ab09-f28babdc1b04-proxy-tls\") pod \"156661a7-3632-4237-ab09-f28babdc1b04\" (UID: \"156661a7-3632-4237-ab09-f28babdc1b04\") "
Jul 02 14:07:54.386969 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.386945    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156661a7-3632-4237-ab09-f28babdc1b04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "156661a7-3632-4237-ab09-f28babdc1b04" (UID: "156661a7-3632-4237-ab09-f28babdc1b04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:07:54.387184 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.387169    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:07:54.387238 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.387186    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "156661a7-3632-4237-ab09-f28babdc1b04" (UID: "156661a7-3632-4237-ab09-f28babdc1b04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:07:54.387238 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.387189    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/156661a7-3632-4237-ab09-f28babdc1b04-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:07:54.388791 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.388768    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156661a7-3632-4237-ab09-f28babdc1b04-kube-api-access-hvv8j" (OuterVolumeSpecName: "kube-api-access-hvv8j") pod "156661a7-3632-4237-ab09-f28babdc1b04" (UID: "156661a7-3632-4237-ab09-f28babdc1b04"). InnerVolumeSpecName "kube-api-access-hvv8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:07:54.388886 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.388831    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156661a7-3632-4237-ab09-f28babdc1b04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "156661a7-3632-4237-ab09-f28babdc1b04" (UID: "156661a7-3632-4237-ab09-f28babdc1b04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:07:54.488067 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.488047    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/156661a7-3632-4237-ab09-f28babdc1b04-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:07:54.488067 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.488068    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/156661a7-3632-4237-ab09-f28babdc1b04-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:07:54.488215 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.488077    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvv8j\" (UniqueName: \"kubernetes.io/projected/156661a7-3632-4237-ab09-f28babdc1b04-kube-api-access-hvv8j\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:07:54.955552 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.955517    2566 generic.go:356] "Generic (PLEG): container finished" podID="156661a7-3632-4237-ab09-f28babdc1b04" containerID="dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42" exitCode=0
Jul 02 14:07:54.955882 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.955591    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"
Jul 02 14:07:54.955882 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.955581    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" event={"ID":"156661a7-3632-4237-ab09-f28babdc1b04","Type":"ContainerDied","Data":"dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42"}
Jul 02 14:07:54.955882 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.955729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc" event={"ID":"156661a7-3632-4237-ab09-f28babdc1b04","Type":"ContainerDied","Data":"f8f59c461558f14d25ac57c8b10da8bb7d1391923164c8ab1890a1821c5fc624"}
Jul 02 14:07:54.955882 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.955759    2566 scope.go:117] "RemoveContainer" containerID="2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c"
Jul 02 14:07:54.964920 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.964904    2566 scope.go:117] "RemoveContainer" containerID="dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42"
Jul 02 14:07:54.972628 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.972606    2566 scope.go:117] "RemoveContainer" containerID="d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3"
Jul 02 14:07:54.975553 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.975529    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"]
Jul 02 14:07:54.979534 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.979510    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e66b7-predictor-769988c997-zrpvc"]
Jul 02 14:07:54.981308 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.981289    2566 scope.go:117] "RemoveContainer" containerID="2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c"
Jul 02 14:07:54.981597 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:07:54.981579    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c\": container with ID starting with 2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c not found: ID does not exist" containerID="2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c"
Jul 02 14:07:54.981680 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.981609    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c"} err="failed to get container status \"2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c\": rpc error: code = NotFound desc = could not find container \"2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c\": container with ID starting with 2a756469908173312d6c1a21949148e814f81b94a0fa2e65508510c139f6966c not found: ID does not exist"
Jul 02 14:07:54.981680 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.981631    2566 scope.go:117] "RemoveContainer" containerID="dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42"
Jul 02 14:07:54.981866 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:07:54.981848    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42\": container with ID starting with dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42 not found: ID does not exist" containerID="dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42"
Jul 02 14:07:54.981933 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.981875    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42"} err="failed to get container status \"dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42\": rpc error: code = NotFound desc = could not find container \"dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42\": container with ID starting with dc7296cd79f88bd5ce0b070a0b86f2eb2bd08f7913f87f728fb0aff72331bf42 not found: ID does not exist"
Jul 02 14:07:54.981933 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.981896    2566 scope.go:117] "RemoveContainer" containerID="d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3"
Jul 02 14:07:54.982151 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:07:54.982131    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3\": container with ID starting with d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3 not found: ID does not exist" containerID="d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3"
Jul 02 14:07:54.982203 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:54.982155    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3"} err="failed to get container status \"d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3\": rpc error: code = NotFound desc = could not find container \"d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3\": container with ID starting with d965d2a86bebf6e584c4e444c0ebb4b161a290008e36e02df4035a5d9507a3a3 not found: ID does not exist"
Jul 02 14:07:55.581636 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.581602    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"]
Jul 02 14:07:55.581940 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.581924    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container"
Jul 02 14:07:55.582004 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.581946    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container"
Jul 02 14:07:55.582004 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.581974    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="storage-initializer"
Jul 02 14:07:55.582004 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.581981    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="storage-initializer"
Jul 02 14:07:55.582004 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.581990    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kube-rbac-proxy"
Jul 02 14:07:55.582004 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.581995    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kube-rbac-proxy"
Jul 02 14:07:55.582265 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.582064    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kserve-container"
Jul 02 14:07:55.582265 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.582073    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="156661a7-3632-4237-ab09-f28babdc1b04" containerName="kube-rbac-proxy"
Jul 02 14:07:55.586607 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.586587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.589161 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.589143    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-bxgtw\""
Jul 02 14:07:55.589291 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.589174    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 14:07:55.589291 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.589148    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:07:55.589442 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.589292    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-9e884-kube-rbac-proxy-sar-config\""
Jul 02 14:07:55.589922 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.589905    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:07:55.590035 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.589959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-9e884-predictor-serving-cert\""
Jul 02 14:07:55.590035 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.590002    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 14:07:55.598261 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.598240    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"]
Jul 02 14:07:55.698150 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.698130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31db5049-4175-4b09-a076-632b10322ecc-proxy-tls\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.698261 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.698158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-cabundle-cert\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.698261 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.698198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6zf\" (UniqueName: \"kubernetes.io/projected/31db5049-4175-4b09-a076-632b10322ecc-kube-api-access-6c6zf\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.698261 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.698227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31db5049-4175-4b09-a076-632b10322ecc-kserve-provision-location\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.698423 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.698285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.799310 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.799285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6zf\" (UniqueName: \"kubernetes.io/projected/31db5049-4175-4b09-a076-632b10322ecc-kube-api-access-6c6zf\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.799423 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.799322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31db5049-4175-4b09-a076-632b10322ecc-kserve-provision-location\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.799423 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.799358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.799423 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.799420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31db5049-4175-4b09-a076-632b10322ecc-proxy-tls\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.799606 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.799449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-cabundle-cert\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.799764 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.799741    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31db5049-4175-4b09-a076-632b10322ecc-kserve-provision-location\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.800066 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.800045    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.800172 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.800146    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-cabundle-cert\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.801852 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.801832    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31db5049-4175-4b09-a076-632b10322ecc-proxy-tls\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.807769 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.807747    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6zf\" (UniqueName: \"kubernetes.io/projected/31db5049-4175-4b09-a076-632b10322ecc-kube-api-access-6c6zf\") pod \"raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:55.897173 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:55.897124    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:07:56.028101 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:56.028079    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"]
Jul 02 14:07:56.030286 ip-10-0-136-2 kubenswrapper[2566]: W0702 14:07:56.030264    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31db5049_4175_4b09_a076_632b10322ecc.slice/crio-94148ed58699a9c98fd6fd28702782200aad830e28ac90dd5b7ed398c43c1c4b WatchSource:0}: Error finding container 94148ed58699a9c98fd6fd28702782200aad830e28ac90dd5b7ed398c43c1c4b: Status 404 returned error can't find the container with id 94148ed58699a9c98fd6fd28702782200aad830e28ac90dd5b7ed398c43c1c4b
Jul 02 14:07:56.504108 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:56.504077    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156661a7-3632-4237-ab09-f28babdc1b04" path="/var/lib/kubelet/pods/156661a7-3632-4237-ab09-f28babdc1b04/volumes"
Jul 02 14:07:56.969996 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:56.969959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" event={"ID":"31db5049-4175-4b09-a076-632b10322ecc","Type":"ContainerStarted","Data":"437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e"}
Jul 02 14:07:56.970151 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:56.970037    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" event={"ID":"31db5049-4175-4b09-a076-632b10322ecc","Type":"ContainerStarted","Data":"94148ed58699a9c98fd6fd28702782200aad830e28ac90dd5b7ed398c43c1c4b"}
Jul 02 14:07:59.985232 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:59.985193    2566 generic.go:356] "Generic (PLEG): container finished" podID="31db5049-4175-4b09-a076-632b10322ecc" containerID="437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e" exitCode=0
Jul 02 14:07:59.985604 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:07:59.985270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" event={"ID":"31db5049-4175-4b09-a076-632b10322ecc","Type":"ContainerDied","Data":"437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e"}
Jul 02 14:08:00.992041 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:00.992007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" event={"ID":"31db5049-4175-4b09-a076-632b10322ecc","Type":"ContainerStarted","Data":"294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66"}
Jul 02 14:08:00.992441 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:00.992068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" event={"ID":"31db5049-4175-4b09-a076-632b10322ecc","Type":"ContainerStarted","Data":"71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d"}
Jul 02 14:08:00.992441 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:00.992337    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:08:00.992602 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:00.992515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:08:00.993515 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:00.993475    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:08:01.035837 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:01.035772    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podStartSLOduration=2.547997621 podStartE2EDuration="6.035759582s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.487761961s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:08:01.034281394 +0000 UTC m=+1380.875918907" watchObservedRunningTime="2026-07-02 14:08:01.035759582 +0000 UTC m=+1380.877397095"
Jul 02 14:08:01.996878 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:01.996832    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:08:07.001658 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:07.001628    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:08:07.002308 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:07.002271    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:08:17.002742 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:17.002694    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:08:27.002740 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:27.002697    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:08:37.002214 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:37.002172    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:08:47.002675 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:47.002624    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:08:57.002792 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:08:57.002699    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:09:07.002278 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:07.002236    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:09:15.499147 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:15.499112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:09:25.717730 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:25.717694    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"]
Jul 02 14:09:25.718203 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:25.717954    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container" containerID="cri-o://71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d" gracePeriod=30
Jul 02 14:09:25.718203 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:25.717986    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kube-rbac-proxy" containerID="cri-o://294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66" gracePeriod=30
Jul 02 14:09:26.363759 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:26.363729    2566 generic.go:356] "Generic (PLEG): container finished" podID="31db5049-4175-4b09-a076-632b10322ecc" containerID="294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66" exitCode=2
Jul 02 14:09:26.363918 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:26.363808    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" event={"ID":"31db5049-4175-4b09-a076-632b10322ecc","Type":"ContainerDied","Data":"294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66"}
Jul 02 14:09:26.997552 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:26.997515    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 02 14:09:29.559556 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.559531    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:09:29.600529 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.600502    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31db5049-4175-4b09-a076-632b10322ecc-proxy-tls\") pod \"31db5049-4175-4b09-a076-632b10322ecc\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") "
Jul 02 14:09:29.600631 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.600552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31db5049-4175-4b09-a076-632b10322ecc-kserve-provision-location\") pod \"31db5049-4175-4b09-a076-632b10322ecc\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") "
Jul 02 14:09:29.600675 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.600627    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-kube-rbac-proxy-sar-config\") pod \"31db5049-4175-4b09-a076-632b10322ecc\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") "
Jul 02 14:09:29.600675 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.600657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-cabundle-cert\") pod \"31db5049-4175-4b09-a076-632b10322ecc\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") "
Jul 02 14:09:29.600756 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.600713    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c6zf\" (UniqueName: \"kubernetes.io/projected/31db5049-4175-4b09-a076-632b10322ecc-kube-api-access-6c6zf\") pod \"31db5049-4175-4b09-a076-632b10322ecc\" (UID: \"31db5049-4175-4b09-a076-632b10322ecc\") "
Jul 02 14:09:29.600835 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.600807    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31db5049-4175-4b09-a076-632b10322ecc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31db5049-4175-4b09-a076-632b10322ecc" (UID: "31db5049-4175-4b09-a076-632b10322ecc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:09:29.600964 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.600943    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31db5049-4175-4b09-a076-632b10322ecc" (UID: "31db5049-4175-4b09-a076-632b10322ecc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:09:29.600964 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.600953    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31db5049-4175-4b09-a076-632b10322ecc" (UID: "31db5049-4175-4b09-a076-632b10322ecc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:09:29.601075 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.601024    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:09:29.601075 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.601044    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31db5049-4175-4b09-a076-632b10322ecc-cabundle-cert\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:09:29.601075 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.601059    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31db5049-4175-4b09-a076-632b10322ecc-kserve-provision-location\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:09:29.602468 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.602439    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31db5049-4175-4b09-a076-632b10322ecc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31db5049-4175-4b09-a076-632b10322ecc" (UID: "31db5049-4175-4b09-a076-632b10322ecc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:09:29.602660 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.602632    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31db5049-4175-4b09-a076-632b10322ecc-kube-api-access-6c6zf" (OuterVolumeSpecName: "kube-api-access-6c6zf") pod "31db5049-4175-4b09-a076-632b10322ecc" (UID: "31db5049-4175-4b09-a076-632b10322ecc"). InnerVolumeSpecName "kube-api-access-6c6zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:09:29.702348 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.702324    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6zf\" (UniqueName: \"kubernetes.io/projected/31db5049-4175-4b09-a076-632b10322ecc-kube-api-access-6c6zf\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:09:29.702348 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:29.702346    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31db5049-4175-4b09-a076-632b10322ecc-proxy-tls\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:09:30.383362 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.383334    2566 generic.go:356] "Generic (PLEG): container finished" podID="31db5049-4175-4b09-a076-632b10322ecc" containerID="71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d" exitCode=0
Jul 02 14:09:30.383520 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.383395    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"
Jul 02 14:09:30.383520 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.383422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" event={"ID":"31db5049-4175-4b09-a076-632b10322ecc","Type":"ContainerDied","Data":"71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d"}
Jul 02 14:09:30.383608 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.383529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x" event={"ID":"31db5049-4175-4b09-a076-632b10322ecc","Type":"ContainerDied","Data":"94148ed58699a9c98fd6fd28702782200aad830e28ac90dd5b7ed398c43c1c4b"}
Jul 02 14:09:30.383608 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.383566    2566 scope.go:117] "RemoveContainer" containerID="294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66"
Jul 02 14:09:30.393198 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.393180    2566 scope.go:117] "RemoveContainer" containerID="71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d"
Jul 02 14:09:30.400554 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.400537    2566 scope.go:117] "RemoveContainer" containerID="437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e"
Jul 02 14:09:30.407555 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.407527    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"]
Jul 02 14:09:30.408302 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.408280    2566 scope.go:117] "RemoveContainer" containerID="294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66"
Jul 02 14:09:30.408655 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:09:30.408633    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66\": container with ID starting with 294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66 not found: ID does not exist" containerID="294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66"
Jul 02 14:09:30.408718 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.408662    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66"} err="failed to get container status \"294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66\": rpc error: code = NotFound desc = could not find container \"294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66\": container with ID starting with 294656e0b518a48bc86e0661199e44e0b388e052fe117f7d1f66506b7b1e1b66 not found: ID does not exist"
Jul 02 14:09:30.408718 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.408678    2566 scope.go:117] "RemoveContainer" containerID="71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d"
Jul 02 14:09:30.408946 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:09:30.408923    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d\": container with ID starting with 71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d not found: ID does not exist" containerID="71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d"
Jul 02 14:09:30.409033 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.408951    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d"} err="failed to get container status \"71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d\": rpc error: code = NotFound desc = could not find container \"71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d\": container with ID starting with 71a3d685b99b11d18142dfdd29e1a2ee619cdd491994b29c245edb4d91a36e2d not found: ID does not exist"
Jul 02 14:09:30.409033 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.408972    2566 scope.go:117] "RemoveContainer" containerID="437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e"
Jul 02 14:09:30.409234 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:09:30.409219    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e\": container with ID starting with 437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e not found: ID does not exist" containerID="437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e"
Jul 02 14:09:30.409291 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.409247    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e"} err="failed to get container status \"437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e\": rpc error: code = NotFound desc = could not find container \"437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e\": container with ID starting with 437a29457f5f259ebf0411597a9beeabecd2cfabb14526a137e2f33e57fd858e not found: ID does not exist"
Jul 02 14:09:30.413392 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.413370    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-9e884-predictor-bf9c64fc7-78b6x"]
Jul 02 14:09:30.503202 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:30.503182    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31db5049-4175-4b09-a076-632b10322ecc" path="/var/lib/kubelet/pods/31db5049-4175-4b09-a076-632b10322ecc/volumes"
Jul 02 14:09:31.515746 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.515714    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c7v68/must-gather-p9bpd"]
Jul 02 14:09:31.516107 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.516044    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="storage-initializer"
Jul 02 14:09:31.516107 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.516055    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="storage-initializer"
Jul 02 14:09:31.516107 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.516075    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kube-rbac-proxy"
Jul 02 14:09:31.516107 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.516081    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kube-rbac-proxy"
Jul 02 14:09:31.516107 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.516093    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container"
Jul 02 14:09:31.516107 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.516099    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container"
Jul 02 14:09:31.516295 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.516154    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kserve-container"
Jul 02 14:09:31.516295 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.516162    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="31db5049-4175-4b09-a076-632b10322ecc" containerName="kube-rbac-proxy"
Jul 02 14:09:31.520977 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.520955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:09:31.523653 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.523628    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7v68\"/\"openshift-service-ca.crt\""
Jul 02 14:09:31.524433 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.524415    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7v68\"/\"kube-root-ca.crt\""
Jul 02 14:09:31.524562 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.524421    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c7v68\"/\"default-dockercfg-trhnn\""
Jul 02 14:09:31.531469 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.531447    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7v68/must-gather-p9bpd"]
Jul 02 14:09:31.616035 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.616013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-must-gather-output\") pod \"must-gather-p9bpd\" (UID: \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\") " pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:09:31.616123 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.616045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjtk\" (UniqueName: \"kubernetes.io/projected/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-kube-api-access-kdjtk\") pod \"must-gather-p9bpd\" (UID: \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\") " pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:09:31.716441 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.716408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjtk\" (UniqueName: \"kubernetes.io/projected/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-kube-api-access-kdjtk\") pod \"must-gather-p9bpd\" (UID: \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\") " pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:09:31.716576 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.716501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-must-gather-output\") pod \"must-gather-p9bpd\" (UID: \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\") " pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:09:31.716766 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.716750    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-must-gather-output\") pod \"must-gather-p9bpd\" (UID: \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\") " pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:09:31.726313 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.726284    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjtk\" (UniqueName: \"kubernetes.io/projected/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-kube-api-access-kdjtk\") pod \"must-gather-p9bpd\" (UID: \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\") " pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:09:31.850599 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.850544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:09:31.983609 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.983581    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7v68/must-gather-p9bpd"]
Jul 02 14:09:31.986653 ip-10-0-136-2 kubenswrapper[2566]: W0702 14:09:31.986623    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4456fc05_ad82_4f2c_af4f_3f6f83f1ac23.slice/crio-4fd68f0a12556a57fe5caeb3c2c496f1c057a0292a68a018bb4b82a9e4803dff WatchSource:0}: Error finding container 4fd68f0a12556a57fe5caeb3c2c496f1c057a0292a68a018bb4b82a9e4803dff: Status 404 returned error can't find the container with id 4fd68f0a12556a57fe5caeb3c2c496f1c057a0292a68a018bb4b82a9e4803dff
Jul 02 14:09:31.988414 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:31.988393    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:09:32.394649 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:32.394616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7v68/must-gather-p9bpd" event={"ID":"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23","Type":"ContainerStarted","Data":"4fd68f0a12556a57fe5caeb3c2c496f1c057a0292a68a018bb4b82a9e4803dff"}
Jul 02 14:09:40.435626 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:40.435022    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7v68/must-gather-p9bpd" event={"ID":"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23","Type":"ContainerStarted","Data":"910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707"}
Jul 02 14:09:40.435626 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:40.435071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7v68/must-gather-p9bpd" event={"ID":"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23","Type":"ContainerStarted","Data":"eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3"}
Jul 02 14:09:40.455252 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:09:40.455200    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c7v68/must-gather-p9bpd" podStartSLOduration=1.975461205 podStartE2EDuration="9.455182181s" totalImagesPullingTime="7.479720976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:09:40.454807567 +0000 UTC m=+1480.296445093" watchObservedRunningTime="2026-07-02 14:09:40.455182181 +0000 UTC m=+1480.296819697"
Jul 02 14:10:01.272609 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:01.272585    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 14:10:01.273059 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:01.272665    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 14:10:01.526327 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:01.526262    2566 generic.go:356] "Generic (PLEG): container finished" podID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" containerID="eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3" exitCode=0
Jul 02 14:10:01.526327 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:01.526298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7v68/must-gather-p9bpd" event={"ID":"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23","Type":"ContainerDied","Data":"eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3"}
Jul 02 14:10:01.526598 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:01.526581    2566 scope.go:117] "RemoveContainer" containerID="eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3"
Jul 02 14:10:01.631379 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:01.631348    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c7v68_must-gather-p9bpd_4456fc05-ad82-4f2c-af4f-3f6f83f1ac23/gather/0.log"
Jul 02 14:10:02.374851 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.374814    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f6j6j/must-gather-69qct"]
Jul 02 14:10:02.378224 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.378206    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f6j6j/must-gather-69qct"
Jul 02 14:10:02.381801 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.381779    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f6j6j\"/\"default-dockercfg-w2289\""
Jul 02 14:10:02.381917 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.381784    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f6j6j\"/\"openshift-service-ca.crt\""
Jul 02 14:10:02.382093 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.382078    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f6j6j\"/\"kube-root-ca.crt\""
Jul 02 14:10:02.393046 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.393021    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f6j6j/must-gather-69qct"]
Jul 02 14:10:02.488073 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.488044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58m9n\" (UniqueName: \"kubernetes.io/projected/ea96518d-b28d-437f-b649-3b8f4575c5fc-kube-api-access-58m9n\") pod \"must-gather-69qct\" (UID: \"ea96518d-b28d-437f-b649-3b8f4575c5fc\") " pod="openshift-must-gather-f6j6j/must-gather-69qct"
Jul 02 14:10:02.488210 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.488080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea96518d-b28d-437f-b649-3b8f4575c5fc-must-gather-output\") pod \"must-gather-69qct\" (UID: \"ea96518d-b28d-437f-b649-3b8f4575c5fc\") " pod="openshift-must-gather-f6j6j/must-gather-69qct"
Jul 02 14:10:02.588751 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.588721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea96518d-b28d-437f-b649-3b8f4575c5fc-must-gather-output\") pod \"must-gather-69qct\" (UID: \"ea96518d-b28d-437f-b649-3b8f4575c5fc\") " pod="openshift-must-gather-f6j6j/must-gather-69qct"
Jul 02 14:10:02.588917 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.588882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58m9n\" (UniqueName: \"kubernetes.io/projected/ea96518d-b28d-437f-b649-3b8f4575c5fc-kube-api-access-58m9n\") pod \"must-gather-69qct\" (UID: \"ea96518d-b28d-437f-b649-3b8f4575c5fc\") " pod="openshift-must-gather-f6j6j/must-gather-69qct"
Jul 02 14:10:02.589184 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.589162    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ea96518d-b28d-437f-b649-3b8f4575c5fc-must-gather-output\") pod \"must-gather-69qct\" (UID: \"ea96518d-b28d-437f-b649-3b8f4575c5fc\") " pod="openshift-must-gather-f6j6j/must-gather-69qct"
Jul 02 14:10:02.605466 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.605434    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58m9n\" (UniqueName: \"kubernetes.io/projected/ea96518d-b28d-437f-b649-3b8f4575c5fc-kube-api-access-58m9n\") pod \"must-gather-69qct\" (UID: \"ea96518d-b28d-437f-b649-3b8f4575c5fc\") " pod="openshift-must-gather-f6j6j/must-gather-69qct"
Jul 02 14:10:02.686717 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.686689    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f6j6j/must-gather-69qct"
Jul 02 14:10:02.825815 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:02.825779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f6j6j/must-gather-69qct"]
Jul 02 14:10:02.829420 ip-10-0-136-2 kubenswrapper[2566]: W0702 14:10:02.829388    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea96518d_b28d_437f_b649_3b8f4575c5fc.slice/crio-345c0a00d61ae772cc09c8e95925993f9e67209872128005ca004678ad97efc0 WatchSource:0}: Error finding container 345c0a00d61ae772cc09c8e95925993f9e67209872128005ca004678ad97efc0: Status 404 returned error can't find the container with id 345c0a00d61ae772cc09c8e95925993f9e67209872128005ca004678ad97efc0
Jul 02 14:10:03.536669 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:03.536629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f6j6j/must-gather-69qct" event={"ID":"ea96518d-b28d-437f-b649-3b8f4575c5fc","Type":"ContainerStarted","Data":"345c0a00d61ae772cc09c8e95925993f9e67209872128005ca004678ad97efc0"}
Jul 02 14:10:04.544421 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:04.544373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f6j6j/must-gather-69qct" event={"ID":"ea96518d-b28d-437f-b649-3b8f4575c5fc","Type":"ContainerStarted","Data":"e9ced45c1b91963fb9669a681f9c31b6e4b3e3f2fbedc1e1e3cb25c0f1075006"}
Jul 02 14:10:04.544940 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:04.544435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f6j6j/must-gather-69qct" event={"ID":"ea96518d-b28d-437f-b649-3b8f4575c5fc","Type":"ContainerStarted","Data":"6a346610c849811cdf498506142b2fd7981d83943451d0db63111b4edd1726d7"}
Jul 02 14:10:04.573073 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:04.573026    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f6j6j/must-gather-69qct" podStartSLOduration=1.756562165 podStartE2EDuration="2.573009248s" totalImagesPullingTime="816.447083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:10:04.571557658 +0000 UTC m=+1504.413195172" watchObservedRunningTime="2026-07-02 14:10:04.573009248 +0000 UTC m=+1504.414646762"
Jul 02 14:10:05.295065 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:05.295041    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5mxmz_4ad7ad64-df1a-46f9-8e19-2b7333b838f5/global-pull-secret-syncer/0.log"
Jul 02 14:10:05.536462 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:05.536434    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wh4zs_0ca65baf-3992-41ea-b4b9-e24681de33d3/konnectivity-agent/0.log"
Jul 02 14:10:05.593554 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:05.593441    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-2.ec2.internal_99227ef48a198d127b2226d0b66cb13b/haproxy/0.log"
Jul 02 14:10:07.158356 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.158312    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-c7v68/must-gather-p9bpd"]
Jul 02 14:10:07.158928 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.158385    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-c7v68/must-gather-p9bpd"]
Jul 02 14:10:07.158928 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.158599    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-c7v68/must-gather-p9bpd" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" containerName="copy" containerID="cri-o://910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707" gracePeriod=2
Jul 02 14:10:07.161871 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:10:07.161839    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p9bpd\" is forbidden: User \"system:node:ip-10-0-136-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c7v68\": no relationship found between node 'ip-10-0-136-2.ec2.internal' and this object" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:10:07.557500 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.552774    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c7v68_must-gather-p9bpd_4456fc05-ad82-4f2c-af4f-3f6f83f1ac23/copy/0.log"
Jul 02 14:10:07.557500 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.556142    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:10:07.559828 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:10:07.559643    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p9bpd\" is forbidden: User \"system:node:ip-10-0-136-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c7v68\": no relationship found between node 'ip-10-0-136-2.ec2.internal' and this object" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:10:07.566988 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.561692    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c7v68_must-gather-p9bpd_4456fc05-ad82-4f2c-af4f-3f6f83f1ac23/copy/0.log"
Jul 02 14:10:07.566988 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.562158    2566 generic.go:356] "Generic (PLEG): container finished" podID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" containerID="910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707" exitCode=143
Jul 02 14:10:07.566988 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.562214    2566 scope.go:117] "RemoveContainer" containerID="910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707"
Jul 02 14:10:07.566988 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.562374    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:10:07.566988 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:10:07.566674    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p9bpd\" is forbidden: User \"system:node:ip-10-0-136-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c7v68\": no relationship found between node 'ip-10-0-136-2.ec2.internal' and this object" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:10:07.588089 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.587245    2566 scope.go:117] "RemoveContainer" containerID="eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3"
Jul 02 14:10:07.620646 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.619223    2566 scope.go:117] "RemoveContainer" containerID="910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707"
Jul 02 14:10:07.620646 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:10:07.619611    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707\": container with ID starting with 910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707 not found: ID does not exist" containerID="910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707"
Jul 02 14:10:07.620646 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.619643    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707"} err="failed to get container status \"910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707\": rpc error: code = NotFound desc = could not find container \"910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707\": container with ID starting with 910ba37668b86ea5fb95b555e31dfc746846384c5575825640adabc3f13d4707 not found: ID does not exist"
Jul 02 14:10:07.620646 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.619679    2566 scope.go:117] "RemoveContainer" containerID="eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3"
Jul 02 14:10:07.620646 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:10:07.619895    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3\": container with ID starting with eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3 not found: ID does not exist" containerID="eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3"
Jul 02 14:10:07.620646 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.619929    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3"} err="failed to get container status \"eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3\": rpc error: code = NotFound desc = could not find container \"eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3\": container with ID starting with eed9d9ca5df06ea030438b96c0214f5b0f2d66de5e2455178b6c3b3b3b0281d3 not found: ID does not exist"
Jul 02 14:10:07.646513 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.643048    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-must-gather-output\") pod \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\" (UID: \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\") "
Jul 02 14:10:07.646513 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.643126    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdjtk\" (UniqueName: \"kubernetes.io/projected/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-kube-api-access-kdjtk\") pod \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\" (UID: \"4456fc05-ad82-4f2c-af4f-3f6f83f1ac23\") "
Jul 02 14:10:07.646513 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.645441    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" (UID: "4456fc05-ad82-4f2c-af4f-3f6f83f1ac23"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:10:07.648651 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.646905    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-kube-api-access-kdjtk" (OuterVolumeSpecName: "kube-api-access-kdjtk") pod "4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" (UID: "4456fc05-ad82-4f2c-af4f-3f6f83f1ac23"). InnerVolumeSpecName "kube-api-access-kdjtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:10:07.744639 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.744553    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjtk\" (UniqueName: \"kubernetes.io/projected/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-kube-api-access-kdjtk\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:10:07.744639 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:07.744613    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23-must-gather-output\") on node \"ip-10-0-136-2.ec2.internal\" DevicePath \"\""
Jul 02 14:10:07.879353 ip-10-0-136-2 kubenswrapper[2566]: E0702 14:10:07.879313    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p9bpd\" is forbidden: User \"system:node:ip-10-0-136-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c7v68\": no relationship found between node 'ip-10-0-136-2.ec2.internal' and this object" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" pod="openshift-must-gather-c7v68/must-gather-p9bpd"
Jul 02 14:10:08.507438 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:08.506954    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" path="/var/lib/kubelet/pods/4456fc05-ad82-4f2c-af4f-3f6f83f1ac23/volumes"
Jul 02 14:10:09.339995 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.339969    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fcc60566-f8a4-4de6-aee3-a433e5090838/alertmanager/0.log"
Jul 02 14:10:09.415843 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.415770    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fcc60566-f8a4-4de6-aee3-a433e5090838/config-reloader/0.log"
Jul 02 14:10:09.486229 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.486198    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fcc60566-f8a4-4de6-aee3-a433e5090838/kube-rbac-proxy-web/0.log"
Jul 02 14:10:09.533212 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.533145    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fcc60566-f8a4-4de6-aee3-a433e5090838/kube-rbac-proxy/0.log"
Jul 02 14:10:09.579273 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.579231    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fcc60566-f8a4-4de6-aee3-a433e5090838/kube-rbac-proxy-metric/0.log"
Jul 02 14:10:09.617953 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.617877    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fcc60566-f8a4-4de6-aee3-a433e5090838/prom-label-proxy/0.log"
Jul 02 14:10:09.654379 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.653956    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fcc60566-f8a4-4de6-aee3-a433e5090838/init-config-reloader/0.log"
Jul 02 14:10:09.713770 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.713745    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-w5g2b_9a4f6553-c51e-49aa-b0b5-dee8541cee26/cluster-monitoring-operator/0.log"
Jul 02 14:10:09.744687 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.744652    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-mnjrf_fd69c4ab-d2a1-416d-af30-b0da27d96123/kube-state-metrics/0.log"
Jul 02 14:10:09.776090 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.776064    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-mnjrf_fd69c4ab-d2a1-416d-af30-b0da27d96123/kube-rbac-proxy-main/0.log"
Jul 02 14:10:09.804838 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.804804    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-mnjrf_fd69c4ab-d2a1-416d-af30-b0da27d96123/kube-rbac-proxy-self/0.log"
Jul 02 14:10:09.921244 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.921164    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dhxp7_0e460137-c685-4475-8d76-a750072a61be/node-exporter/0.log"
Jul 02 14:10:09.949335 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.949302    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dhxp7_0e460137-c685-4475-8d76-a750072a61be/kube-rbac-proxy/0.log"
Jul 02 14:10:09.975859 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:09.975769    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dhxp7_0e460137-c685-4475-8d76-a750072a61be/init-textfile/0.log"
Jul 02 14:10:10.190749 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.190719    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-sfs24_3267367b-c780-49f0-8489-40fdf391a16b/kube-rbac-proxy-main/0.log"
Jul 02 14:10:10.217123 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.217083    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-sfs24_3267367b-c780-49f0-8489-40fdf391a16b/kube-rbac-proxy-self/0.log"
Jul 02 14:10:10.244776 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.244747    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-sfs24_3267367b-c780-49f0-8489-40fdf391a16b/openshift-state-metrics/0.log"
Jul 02 14:10:10.305830 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.305798    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed426926-6db7-489c-8836-020e215de3ef/prometheus/0.log"
Jul 02 14:10:10.329145 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.329111    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed426926-6db7-489c-8836-020e215de3ef/config-reloader/0.log"
Jul 02 14:10:10.357832 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.357801    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed426926-6db7-489c-8836-020e215de3ef/thanos-sidecar/0.log"
Jul 02 14:10:10.383410 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.383373    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed426926-6db7-489c-8836-020e215de3ef/kube-rbac-proxy-web/0.log"
Jul 02 14:10:10.416778 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.416744    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed426926-6db7-489c-8836-020e215de3ef/kube-rbac-proxy/0.log"
Jul 02 14:10:10.446234 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.446157    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed426926-6db7-489c-8836-020e215de3ef/kube-rbac-proxy-thanos/0.log"
Jul 02 14:10:10.472789 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.472760    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed426926-6db7-489c-8836-020e215de3ef/init-config-reloader/0.log"
Jul 02 14:10:10.567522 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.567467    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-v48zk_0b6446cb-36a3-44d4-a51e-21547a9186c0/prometheus-operator-admission-webhook/0.log"
Jul 02 14:10:10.689456 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.689424    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb4f5765f-rlkt6_3cc8d4be-b925-4fb5-a416-063d4ba8c9a6/thanos-query/0.log"
Jul 02 14:10:10.715818 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.715734    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb4f5765f-rlkt6_3cc8d4be-b925-4fb5-a416-063d4ba8c9a6/kube-rbac-proxy-web/0.log"
Jul 02 14:10:10.751826 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.751789    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb4f5765f-rlkt6_3cc8d4be-b925-4fb5-a416-063d4ba8c9a6/kube-rbac-proxy/0.log"
Jul 02 14:10:10.777711 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.777680    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb4f5765f-rlkt6_3cc8d4be-b925-4fb5-a416-063d4ba8c9a6/prom-label-proxy/0.log"
Jul 02 14:10:10.812920 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.812888    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb4f5765f-rlkt6_3cc8d4be-b925-4fb5-a416-063d4ba8c9a6/kube-rbac-proxy-rules/0.log"
Jul 02 14:10:10.853079 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:10.852981    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cb4f5765f-rlkt6_3cc8d4be-b925-4fb5-a416-063d4ba8c9a6/kube-rbac-proxy-metrics/0.log"
Jul 02 14:10:12.109292 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:12.109189    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-wx95f_5cd8ea55-fb58-4a20-ad89-fbbb017846fd/networking-console-plugin/0.log"
Jul 02 14:10:13.102800 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:13.102775    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-n7nq7_0a0dfc85-0d34-4338-a559-553b52c97cd2/download-server/0.log"
Jul 02 14:10:13.603191 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:13.603161    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-69d9f679db-cl6t9_932531fc-a0e6-4e3a-a5ea-46a1357a6409/volume-data-source-validator/0.log"
Jul 02 14:10:14.341425 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.341390    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"]
Jul 02 14:10:14.341970 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.341950    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" containerName="gather"
Jul 02 14:10:14.342027 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.341975    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" containerName="gather"
Jul 02 14:10:14.342027 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.341994    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" containerName="copy"
Jul 02 14:10:14.342027 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.342004    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" containerName="copy"
Jul 02 14:10:14.342152 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.342110    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" containerName="copy"
Jul 02 14:10:14.342152 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.342128    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4456fc05-ad82-4f2c-af4f-3f6f83f1ac23" containerName="gather"
Jul 02 14:10:14.346307 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.346286    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.356505 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.356468    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"]
Jul 02 14:10:14.409960 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.409922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmhh5\" (UniqueName: \"kubernetes.io/projected/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-kube-api-access-mmhh5\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.410145 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.409988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-podres\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.410145 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.410083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-lib-modules\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.410145 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.410135    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-sys\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.410340 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.410159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-proc\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.510947 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.510912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-podres\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.511125 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.510995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-lib-modules\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.511125 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.511042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-sys\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.511125 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.511064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-proc\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.511125 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.511104    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-podres\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.511345 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.511130    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-lib-modules\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.511345 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.511151    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-sys\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.511345 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.511158    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-proc\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.511345 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.511176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmhh5\" (UniqueName: \"kubernetes.io/projected/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-kube-api-access-mmhh5\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.521618 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.521590    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmhh5\" (UniqueName: \"kubernetes.io/projected/8cec263c-18ee-4fa9-8dcc-3e66c431e5eb-kube-api-access-mmhh5\") pod \"perf-node-gather-daemonset-dn8nm\" (UID: \"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb\") " pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.584313 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.584271    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qkjj8_aedca59b-0e89-4e5b-9092-3628bfc55297/dns/0.log"
Jul 02 14:10:14.608869 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.608804    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qkjj8_aedca59b-0e89-4e5b-9092-3628bfc55297/kube-rbac-proxy/0.log"
Jul 02 14:10:14.659801 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.659778    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:14.797739 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.797697    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rq799_2a804050-c3a7-4cf3-923b-5146f3dde7c3/dns-node-resolver/0.log"
Jul 02 14:10:14.807343 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:14.807315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"]
Jul 02 14:10:14.810795 ip-10-0-136-2 kubenswrapper[2566]: W0702 14:10:14.810770    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8cec263c_18ee_4fa9_8dcc_3e66c431e5eb.slice/crio-f5178fdbdad0a2a4a72ff5e707ff3ce39cbf90133f5c74400f71d74ca933aa23 WatchSource:0}: Error finding container f5178fdbdad0a2a4a72ff5e707ff3ce39cbf90133f5c74400f71d74ca933aa23: Status 404 returned error can't find the container with id f5178fdbdad0a2a4a72ff5e707ff3ce39cbf90133f5c74400f71d74ca933aa23
Jul 02 14:10:15.366854 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:15.366811    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnjj7_095b3c78-4147-4098-b517-722b1e4941f1/node-ca/0.log"
Jul 02 14:10:15.609425 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:15.609387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm" event={"ID":"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb","Type":"ContainerStarted","Data":"fc28b6bd231da5d0acea2be005d9a12abc57f2abb2ceb718b42a3e5b816cad07"}
Jul 02 14:10:15.609909 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:15.609450    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm" event={"ID":"8cec263c-18ee-4fa9-8dcc-3e66c431e5eb","Type":"ContainerStarted","Data":"f5178fdbdad0a2a4a72ff5e707ff3ce39cbf90133f5c74400f71d74ca933aa23"}
Jul 02 14:10:15.609909 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:15.609664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:16.541014 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:16.540982    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2ljkk_8eaabca2-d6be-46b6-9b5d-d7db2490f413/serve-healthcheck-canary/0.log"
Jul 02 14:10:17.114417 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:17.114385    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jshgg_a1d430fe-cf31-42f5-b780-28352f84d9dd/kube-rbac-proxy/0.log"
Jul 02 14:10:17.138550 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:17.138517    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jshgg_a1d430fe-cf31-42f5-b780-28352f84d9dd/exporter/0.log"
Jul 02 14:10:17.164615 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:17.164591    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jshgg_a1d430fe-cf31-42f5-b780-28352f84d9dd/extractor/0.log"
Jul 02 14:10:19.649183 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:19.649156    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kmlj8_b2612dae-1721-4e5d-bdbf-b419d77189ec/s3-init/0.log"
Jul 02 14:10:19.687975 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:19.687950    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kmlj8_b2612dae-1721-4e5d-bdbf-b419d77189ec/download-hf-model/0.log"
Jul 02 14:10:21.625225 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:21.625199    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm"
Jul 02 14:10:21.646936 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:21.646457    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f6j6j/perf-node-gather-daemonset-dn8nm" podStartSLOduration=7.646437617 podStartE2EDuration="7.646437617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:10:15.627781138 +0000 UTC m=+1515.469418654" watchObservedRunningTime="2026-07-02 14:10:21.646437617 +0000 UTC m=+1521.488075132"
Jul 02 14:10:24.271320 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:24.271279    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-v7nvl_e3d9299e-86c3-4cf4-9464-6a3cb72e0287/kube-storage-version-migrator-operator/1.log"
Jul 02 14:10:24.273120 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:24.273081    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-v7nvl_e3d9299e-86c3-4cf4-9464-6a3cb72e0287/kube-storage-version-migrator-operator/0.log"
Jul 02 14:10:25.208928 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.208899    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-94mkh_a40daca0-6d80-4669-b2de-895ea47b7087/kube-multus/0.log"
Jul 02 14:10:25.433316 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.433289    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hzt5_04a7dc11-0af2-4222-b6bf-06c30c9f50c0/kube-multus-additional-cni-plugins/0.log"
Jul 02 14:10:25.458411 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.458389    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hzt5_04a7dc11-0af2-4222-b6bf-06c30c9f50c0/egress-router-binary-copy/0.log"
Jul 02 14:10:25.485186 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.485100    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hzt5_04a7dc11-0af2-4222-b6bf-06c30c9f50c0/cni-plugins/0.log"
Jul 02 14:10:25.510855 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.510829    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hzt5_04a7dc11-0af2-4222-b6bf-06c30c9f50c0/bond-cni-plugin/0.log"
Jul 02 14:10:25.537909 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.537885    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hzt5_04a7dc11-0af2-4222-b6bf-06c30c9f50c0/routeoverride-cni/0.log"
Jul 02 14:10:25.567073 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.567047    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hzt5_04a7dc11-0af2-4222-b6bf-06c30c9f50c0/whereabouts-cni-bincopy/0.log"
Jul 02 14:10:25.594157 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.594132    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hzt5_04a7dc11-0af2-4222-b6bf-06c30c9f50c0/whereabouts-cni/0.log"
Jul 02 14:10:25.852681 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.852601    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dm4g2_66cf39f9-4496-49ad-aabc-e9453dab385a/network-metrics-daemon/0.log"
Jul 02 14:10:25.876442 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:25.876413    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dm4g2_66cf39f9-4496-49ad-aabc-e9453dab385a/kube-rbac-proxy/0.log"
Jul 02 14:10:27.017527 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:27.017497    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-controller/0.log"
Jul 02 14:10:27.037972 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:27.037944    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/1.log"
Jul 02 14:10:27.038139 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:27.037977    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovn-acl-logging/0.log"
Jul 02 14:10:27.065228 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:27.065205    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/kube-rbac-proxy-node/0.log"
Jul 02 14:10:27.099024 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:27.098999    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 14:10:27.132348 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:27.132319    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/northd/0.log"
Jul 02 14:10:27.158639 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:27.158614    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/nbdb/0.log"
Jul 02 14:10:27.190284 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:27.190256    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/sbdb/0.log"
Jul 02 14:10:27.258428 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:27.258398    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j86jl_5a8440fa-890d-46a8-ae75-c0c25838ebe4/ovnkube-controller/0.log"
Jul 02 14:10:28.635646 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:28.635620    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zn2d9_d5ada156-013a-43b9-a444-d4fd7984a5be/network-check-target-container/0.log"
Jul 02 14:10:29.591446 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:29.591416    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-6nmmk_a780d76b-930b-46f0-96c7-ffab4bc1166a/iptables-alerter/0.log"
Jul 02 14:10:30.285030 ip-10-0-136-2 kubenswrapper[2566]: I0702 14:10:30.285006    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-fpdcn_31be0683-8d85-4b98-927f-7161f9326721/tuned/0.log"