Jul 07 14:07:24.145526 ip-10-0-135-29 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 14:07:24.145538 ip-10-0-135-29 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 14:07:24.145547 ip-10-0-135-29 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 14:07:24.145775 ip-10-0-135-29 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 14:07:34.289415 ip-10-0-135-29 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 14:07:34.289433 ip-10-0-135-29 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot df647ffa168844c2b96f5670450423e0 --
Jul 07 14:09:12.659682 ip-10-0-135-29 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.097375    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103654    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103681    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103688    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103693    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103703    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103707    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:13.212238 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103713    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103718    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103723    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103727    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103730    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103734    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103738    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103742    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103746    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103750    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103754    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103758    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103761    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103765    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103769    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103773    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103778    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103782    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103786    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103790    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:13.213393 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103795    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:13.182394 ip-10-0-135-29 systemd[1]: Started Kubernetes Kubelet.
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103799    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103802    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103806    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103810    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103814    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103818    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103823    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103828    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103834    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103838    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103844    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103849    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103852    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103856    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103860    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103864    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103872    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103876    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:13.215262 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103879    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103883    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103886    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103891    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103895    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103898    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103902    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103906    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103910    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103914    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103918    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103922    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103926    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103930    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103934    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103938    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103942    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103947    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103952    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103955    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:13.216697 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103959    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103963    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103967    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103970    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103975    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103980    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103983    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103987    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103991    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103995    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.103999    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104002    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104006    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104010    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104014    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104017    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104022    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104026    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104029    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104033    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:13.219024 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104036    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104042    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104046    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104049    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104053    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104057    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104061    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104066    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104069    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104073    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104077    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104081    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104084    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.104088    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105160    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105170    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105173    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105177    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105182    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105187    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:13.221035 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105191    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105195    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105201    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105207    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105211    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105215    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105221    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105227    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105231    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105236    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105240    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105244    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105248    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105253    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105257    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105261    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105266    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105274    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105278    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:13.222948 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105282    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105286    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105290    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105294    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105298    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105302    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105306    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105309    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105313    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105317    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105320    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105324    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105353    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105358    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105361    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105365    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105376    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105380    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105384    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105387    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:13.225190 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105392    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105396    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105399    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105403    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105407    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105411    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105415    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105419    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105424    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105428    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105455    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105461    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105467    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105472    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105476    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105480    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105486    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105490    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105494    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105498    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:13.233831 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105502    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105506    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105510    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105514    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105520    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105524    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105529    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105532    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105536    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105540    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105544    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105547    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105551    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105554    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105558    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105562    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105566    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105569    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105573    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105577    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:13.234500 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105581    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105584    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105590    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105594    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105597    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105601    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105605    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105609    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105613    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105616    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105620    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105624    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105627    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.105631    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106201    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106209    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106217    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106223    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106226    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106231    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:13.235127 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106235    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106239    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106242    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106246    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106249    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106254    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106258    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106261    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106265    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106269    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106273    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106277    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106281    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106287    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106294    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106300    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106305    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106309    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106313    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:13.235787 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106317    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106321    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106325    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106328    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106349    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106353    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106358    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106364    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106369    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106374    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106378    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106383    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106387    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106391    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106395    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106399    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106403    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106406    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106410    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:13.236305 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106414    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106418    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106421    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106425    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106429    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106433    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106436    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106440    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106445    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106449    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106452    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106456    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106459    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106463    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106467    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106471    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106474    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106478    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106482    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106485    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:13.236881 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106489    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106492    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106497    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106501    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106505    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106511    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106515    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106519    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106523    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106527    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106531    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106535    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106538    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106542    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106546    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106550    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106553    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106556    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106560    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:13.237446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106564    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106570    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106573    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106577    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106582    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106586    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106589    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106593    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106597    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106601    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106604    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106608    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106612    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106616    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106620    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.106624    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106723    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106734    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106741    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106749    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106754    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 14:09:13.238019 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106759    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106766    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106772    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106777    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106782    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106787    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106791    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106796    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106801    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106805    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106810    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106814    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106823    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106852    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106857    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106862    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106867    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106872    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106887    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106892    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106896    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106901    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106906    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106911    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 14:09:13.238598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106915    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106920    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106926    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106932    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106936    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106940    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106944    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106950    2569 flags.go:64] FLAG: --enable-server="true"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106954    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106960    2569 flags.go:64] FLAG: --event-burst="100"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106965    2569 flags.go:64] FLAG: --event-qps="50"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106970    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106974    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106979    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106986    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106990    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106994    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.106999    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107002    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107007    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107013    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107017    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107022    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107026    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107031    2569 flags.go:64] FLAG: --feature-gates=""
Jul 07 14:09:13.239252 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107037    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107041    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107046    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107050    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107054    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107059    2569 flags.go:64] FLAG: --help="false"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107063    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107068    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107072    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107079    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107084    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107089    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107093    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107098    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107102    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107107    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107112    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107117    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107121    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107125    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107129    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107134    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107138    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107142    2569 flags.go:64] FLAG: --lock-file=""
Jul 07 14:09:13.240063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107146    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107151    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107155    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107165    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107169    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107173    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107178    2569 flags.go:64] FLAG: --logging-format="text"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107182    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107187    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107192    2569 flags.go:64] FLAG: --manifest-url=""
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107196    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107202    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107207    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107213    2569 flags.go:64] FLAG: --max-pods="110"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107218    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107223    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107228    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107234    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107239    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107243    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107247    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107258    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107264    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107268    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 14:09:13.240812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107277    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107282    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107290    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107324    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107346    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107351    2569 flags.go:64] FLAG: --port="10250"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107357    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107361    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0368fdb8b2332321d"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107365    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107370    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107374    2569 flags.go:64] FLAG: --register-node="true"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107380    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107386    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107391    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107395    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107399    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107405    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107410    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107415    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107419    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107425    2569 flags.go:64] FLAG: --runonce="false"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107429    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107434    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107438    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107443    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 14:09:13.243866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107447    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107453    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107457    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107462    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107466    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107470    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107475    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107479    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107485    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107489    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107493    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107501    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107506    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107511    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107517    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107521    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107525    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107529    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107535    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107540    2569 flags.go:64] FLAG: --v="2"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107546    2569 flags.go:64] FLAG: --version="false"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107551    2569 flags.go:64] FLAG: --vmodule=""
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107556    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.107561    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107743    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:13.244979 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107749    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107754    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107757    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107762    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107766    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107769    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107773    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107776    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107782    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107785    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107790    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107794    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107798    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107802    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107806    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107809    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107814    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107818    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107822    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107825    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:13.402461 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107829    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107833    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107837    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107841    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107845    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107851    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107854    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107858    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107861    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107865    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107870    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107873    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107877    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107881    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107885    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107888    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107892    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107896    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107900    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107904    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:13.422501 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107908    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107912    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107916    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107920    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107924    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107927    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107931    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107935    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107940    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107946    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107952    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107956    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107959    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107963    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107967    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107971    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107974    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107980    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107984    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107988    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:13.423486 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107992    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107996    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.107999    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108003    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108007    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108011    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108015    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108019    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108023    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108026    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108030    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108034    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108040    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108044    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108047    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108051    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108055    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108059    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108062    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:13.424233 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108066    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108070    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108074    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108078    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108083    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108089    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108093    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108096    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108100    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108103    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108109    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108113    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108117    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108121    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108124    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108130    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108135    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108140    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:13.424838 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.108144    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.108922    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.117986    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.118012    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.118036    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.118041    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118118    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118125    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118129    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118132    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118135    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118138    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118141    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118145    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118148    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:13.425427 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118152    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118155    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118158    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118161    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118163    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118166    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118169    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118172    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118175    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118181    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118184    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118186    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118189    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118192    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118195    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118198    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118201    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118216    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118219    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118222    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:13.426045 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118224    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118227    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118231    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118233    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118236    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118239    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118242    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118246    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118250    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118253    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118256    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118259    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118262    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118264    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118268    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118271    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118275    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118280    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118282    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:13.426779 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118285    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118288    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118292    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118295    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118299    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118301    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118304    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118307    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118310    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118313    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118316    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118319    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118322    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118325    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118328    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118347    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118350    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118353    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118356    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118358    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:13.427457 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118361    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118364    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118367    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118369    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118372    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118374    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118377    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118384    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118386    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118389    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118392    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118396    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118399    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118402    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118406    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118409    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118411    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118414    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118416    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118419    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:13.428154 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118421    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118424    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118427    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118429    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118432    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118435    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118437    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118440    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118442    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118445    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118449    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.118455    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118606    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118611    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118614    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:13.428878 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118617    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118620    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118622    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118625    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118629    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118632    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118634    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118637    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118639    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118645    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118650    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118654    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118658    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118660    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118663    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118666    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118669    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118671    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118674    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:13.429421 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118677    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118679    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118682    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118685    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118688    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118690    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118693    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118696    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118698    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118701    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118704    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118706    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118709    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118712    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118714    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118717    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118720    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118724    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118727    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:13.430308 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118729    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118732    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118734    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118737    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118740    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118743    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118747    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118749    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118752    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118754    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118757    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118759    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118762    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118764    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118767    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118769    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118772    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118774    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118777    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118779    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:13.430934 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118782    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118785    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118787    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118790    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118792    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118795    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118797    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118800    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118802    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118804    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118808    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118811    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118813    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118816    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118819    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118823    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118825    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118829    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118833    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118835    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:13.431563 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118838    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118840    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118843    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118845    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118848    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118850    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118853    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118857    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118860    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118863    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118866    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118869    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118871    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118874    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118876    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118879    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118881    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:13.432160 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:13.118884    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.118888    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.119038    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.122537    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.124303    2569 server.go:1019] "Starting client certificate rotation"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.124441    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.125261    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.127262    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.128518    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.138379    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.143250    2569 log.go:25] "Validated CRI v1 image API"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.145172    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 14:09:13.432765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.147956    2569 fs.go:135] Filesystem UUIDs: map[4676c8d9-057d-4c97-8794-5e7a3863311d:/dev/nvme0n1p3 5153d4ef-24a6-495e-81c8-46f65a7df8aa:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 07 14:09:13.433167 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.147975    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 14:09:13.433580 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.154005    2569 manager.go:216] Machine: {Timestamp:2026-07-07 14:09:13.153500676 +0000 UTC m=+0.378589020 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099050 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25c7bce6497c5ad680c2b1991c2371 SystemUUID:ec25c7bc-e649-7c5a-d680-c2b1991c2371 BootID:df647ffa-1688-44c2-b96f-5670450423e0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dc:68:27:c7:1b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dc:68:27:c7:1b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:1c:6a:eb:fd:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 14:09:13.433580 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.154122    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 14:09:13.433580 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.154217    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 14:09:13.433580 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.158083    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 14:09:13.433580 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.158139    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 14:09:13.433580 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.158999    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 14:09:13.433580 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.159010    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 14:09:13.433580 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.159018    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.159707    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.159871    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.161795    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.166620    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.166782    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.166818    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.168694    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.168757    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.168770    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.168838    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.169949    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.169974    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.172535    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.175122    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.175152    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177254    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177272    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177278    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177283    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177293    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177299    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177307    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177312    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177319    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177325    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177344    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.177350    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.178171    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.178182    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.181507    2569 server.go:1268] "Started kubelet"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.181564    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.181632    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.181708    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.182683    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.182870    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.182877    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.183885    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 14:09:13.433972 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.183937    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.186546    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-29.ec2.internal.18c0072f88f75309  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-29.ec2.internal,UID:ip-10-0-135-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-29.ec2.internal,},FirstTimestamp:2026-07-07 14:09:13.181483785 +0000 UTC m=+0.406572129,LastTimestamp:2026-07-07 14:09:13.181483785 +0000 UTC m=+0.406572129,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-29.ec2.internal,}"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.190037    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.194459    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.194470    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.194549    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.195328    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.195369    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.195382    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.195519    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.195529    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.195668    2569 factory.go:55] Registering systemd factory
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.195708    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.195973    2569 factory.go:153] Registering CRI-O factory
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.195991    2569 factory.go:223] Registration of the crio container factory successfully
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.196053    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.196086    2569 factory.go:103] Registering Raw factory
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.196106    2569 manager.go:1196] Started watching for new ooms in manager
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.196569    2569 manager.go:349] Starting recovery of all containers
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.197396    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.197530    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.210295    2569 manager.go:354] Recovery completed
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.215910    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.218025    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.218078    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.218089    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.218733    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.218748    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.218755    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.218767    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.219515    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.219540    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.219549    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:13.435613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.221004    2569 policy_none.go:49] "None policy: Start"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.221216    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.221232    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.221383    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8wxw4"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.223376    2569 policy_none.go:47] "Start"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.228354    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-29.ec2.internal.18c0072f8b256af0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-29.ec2.internal,UID:ip-10-0-135-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-29.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-29.ec2.internal,},FirstTimestamp:2026-07-07 14:09:13.218058992 +0000 UTC m=+0.443147336,LastTimestamp:2026-07-07 14:09:13.218058992 +0000 UTC m=+0.443147336,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-29.ec2.internal,}"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.230076    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8wxw4"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.272209    2569 manager.go:337] "Starting Device Plugin manager"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.272251    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.272264    2569 server.go:87] "Starting device plugin registration server"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.272620    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.272638    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.272973    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.273049    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.273057    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.274853    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.274895    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.316571    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.317836    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.317877    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.317903    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.317961    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.320507    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.373572    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.374650    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.374684    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.374699    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.374724    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.381955    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.381980    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-29.ec2.internal\": node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.418351    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"]
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.418452    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:13.436597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.419635    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.419663    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.419674    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.420939    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.422479    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.422668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.422713    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.423364    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.423391    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.423366    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.423428    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.423400    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.423465    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.425841    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.425875    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.426745    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.426771    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.426799    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.454083    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-29.ec2.internal\" not found" node="ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.457822    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-29.ec2.internal\" not found" node="ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.497404    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/479338554c92fbd8bd392d8bacd28b2d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-29.ec2.internal\" (UID: \"479338554c92fbd8bd392d8bacd28b2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.497437    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a2d38a3dcfd4cfda671d94785cdebdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal\" (UID: \"0a2d38a3dcfd4cfda671d94785cdebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.497463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a2d38a3dcfd4cfda671d94785cdebdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal\" (UID: \"0a2d38a3dcfd4cfda671d94785cdebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.521734    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:13.598595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.597820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/479338554c92fbd8bd392d8bacd28b2d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-29.ec2.internal\" (UID: \"479338554c92fbd8bd392d8bacd28b2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.599467 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.597853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a2d38a3dcfd4cfda671d94785cdebdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal\" (UID: \"0a2d38a3dcfd4cfda671d94785cdebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.599467 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.597871    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a2d38a3dcfd4cfda671d94785cdebdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal\" (UID: \"0a2d38a3dcfd4cfda671d94785cdebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.599467 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.597916    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/479338554c92fbd8bd392d8bacd28b2d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-29.ec2.internal\" (UID: \"479338554c92fbd8bd392d8bacd28b2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.599467 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.597982    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a2d38a3dcfd4cfda671d94785cdebdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal\" (UID: \"0a2d38a3dcfd4cfda671d94785cdebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:13.599467 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:13.597937    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a2d38a3dcfd4cfda671d94785cdebdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal\" (UID: \"0a2d38a3dcfd4cfda671d94785cdebdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:14.853295 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:13.621871    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:14.853295 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:14.838761    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:14.941856 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:14.876103    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:14.941856 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:14.878420    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:14.941856 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:14.939070    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:15.143702 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:15.039790    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:15.143702 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:15.140226    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:15.298451 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:15.240704    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:15.298451 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.240709    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 14:09:15.298451 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.240902    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 14:09:15.444799 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:15.340862    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:15.444799 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.343525    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:15.444799 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.349614    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:04:14 +0000 UTC" deadline="2028-03-26 04:30:13.104624846 +0000 UTC"
Jul 07 14:09:15.444799 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.349648    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15062h20m57.754979896s"
Jul 07 14:09:15.444799 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.403964    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:15.444799 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.405267    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:15.444799 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.405296    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:15.444799 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.405309    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:15.445963 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:15.445798    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:15.553398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.500522    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:15.553398 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:15.515333    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479338554c92fbd8bd392d8bacd28b2d.slice/crio-b492ff5d40bb9cdfe554d304a66642715543cf9a54aa2cfe55b6142763a50120 WatchSource:0}: Error finding container b492ff5d40bb9cdfe554d304a66642715543cf9a54aa2cfe55b6142763a50120: Status 404 returned error can't find the container with id b492ff5d40bb9cdfe554d304a66642715543cf9a54aa2cfe55b6142763a50120
Jul 07 14:09:15.553398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.524781    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:09:15.553398 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:15.525171    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2d38a3dcfd4cfda671d94785cdebdd.slice/crio-56ee231e74c608d895992947889c6c89ac36b647e128bb797a4c12cbf84936bc WatchSource:0}: Error finding container 56ee231e74c608d895992947889c6c89ac36b647e128bb797a4c12cbf84936bc: Status 404 returned error can't find the container with id 56ee231e74c608d895992947889c6c89ac36b647e128bb797a4c12cbf84936bc
Jul 07 14:09:15.553398 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:15.546441    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:15.661647 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:15.647006    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-29.ec2.internal\" not found"
Jul 07 14:09:15.670013 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.669718    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:15.759327 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.711845    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:15.759327 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.721656    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:09:15.759327 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.722543    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal"
Jul 07 14:09:15.759327 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:15.736664    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:09:16.287596 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.287502    2569 apiserver.go:52] "Watching apiserver"
Jul 07 14:09:16.294419 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.294240    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 14:09:16.296344 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.296094    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-xzqcm","kube-system/global-pull-secret-syncer-zhwzv","kube-system/konnectivity-agent-lffsb","openshift-image-registry/node-ca-n5pz8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal","openshift-multus/multus-additional-cni-plugins-4mkfq","openshift-multus/network-metrics-daemon-8jm6t","openshift-network-operator/iptables-alerter-f62fq","kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988","openshift-cluster-node-tuning-operator/tuned-t6zkp","openshift-dns/node-resolver-cfxrl","openshift-multus/multus-785hx","openshift-network-diagnostics/network-check-target-9r2vh"]
Jul 07 14:09:16.299047 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.298692    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:16.299047 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.298793    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:16.300961 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.300933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.303130 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.303090    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:09:16.303258 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.303098    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 14:09:16.303734 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.303366    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:16.303734 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.303421    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7nzhv\""
Jul 07 14:09:16.305561 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.305370    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.305728 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.305704    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 14:09:16.305979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.305962    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kvpc9\""
Jul 07 14:09:16.306066 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.306031    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 14:09:16.307494 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.307278    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 14:09:16.307912 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.307635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:16.307912 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.307687    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 14:09:16.307912 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.307725    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:16.307912 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.307736    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqxzk\""
Jul 07 14:09:16.307912 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.307871    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 14:09:16.310251 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.310231    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.312220 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.312199    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.312333 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.312284    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 14:09:16.312611 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.312593    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 14:09:16.312767 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.312750    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 14:09:16.312829 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.312793    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 14:09:16.312829 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.312823    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 14:09:16.312987 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.312970    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 14:09:16.313044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.312986    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pb54n\""
Jul 07 14:09:16.314220 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.314103    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:09:16.314653 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.314628    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 14:09:16.314782 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.314757    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-82hgx\""
Jul 07 14:09:16.314871 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.314857    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 14:09:16.315087 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.315050    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.317013 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.316989    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:16.317170 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.317066    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:16.317265 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.317244    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 14:09:16.317329 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.317317    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z44n9\""
Jul 07 14:09:16.321008 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.320271    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.322415 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.322379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cab4996-d1b9-4425-ba8a-c5889b50acfb-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.324255 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cab4996-d1b9-4425-ba8a-c5889b50acfb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.324396 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-run-multus-certs\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.324396 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324328    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:16.324396 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-run\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.324396 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324387    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-lib-modules\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.324605 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324431    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70208e8b-d198-4a27-abb8-248b9a6b9234-serviceca\") pod \"node-ca-n5pz8\" (UID: \"70208e8b-d198-4a27-abb8-248b9a6b9234\") " pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.324605 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324461    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7zn7\" (UniqueName: \"kubernetes.io/projected/70208e8b-d198-4a27-abb8-248b9a6b9234-kube-api-access-k7zn7\") pod \"node-ca-n5pz8\" (UID: \"70208e8b-d198-4a27-abb8-248b9a6b9234\") " pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.324605 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324493    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-multus-socket-dir-parent\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.324605 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-var-lib-kubelet\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.324605 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-hostroot\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.324605 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324554    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 14:09:16.324605 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65a0ecad-5ba4-47c7-879d-3fd9aae552fa-agent-certs\") pod \"konnectivity-agent-lffsb\" (UID: \"65a0ecad-5ba4-47c7-879d-3fd9aae552fa\") " pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-kubernetes\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-sysctl-d\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-cnibin\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-multus-cni-dir\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324736    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-cnibin\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324773    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-os-release\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324801    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324805    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-etc-kubernetes\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-sysctl-conf\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-host\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324896    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70208e8b-d198-4a27-abb8-248b9a6b9234-host\") pod \"node-ca-n5pz8\" (UID: \"70208e8b-d198-4a27-abb8-248b9a6b9234\") " pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.324934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324927    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cab4996-d1b9-4425-ba8a-c5889b50acfb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.324968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7bp7\" (UniqueName: \"kubernetes.io/projected/4298379e-f6c7-44eb-b922-f393e007c4af-kube-api-access-h7bp7\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/74690fa1-9808-4516-81d4-6428a970246d-iptables-alerter-script\") pod \"iptables-alerter-f62fq\" (UID: \"74690fa1-9808-4516-81d4-6428a970246d\") " pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325062    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325071    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfzx5\" (UniqueName: \"kubernetes.io/projected/41858fe6-7f55-456d-b7f7-59470406c3cf-kube-api-access-wfzx5\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325105    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325159    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4xpg\" (UniqueName: \"kubernetes.io/projected/6cab4996-d1b9-4425-ba8a-c5889b50acfb-kube-api-access-k4xpg\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325189    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0933ede-8054-466c-82a5-83ad1751743b-cni-binary-copy\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-run-k8s-cni-cncf-io\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-run-netns\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-multus-conf-dir\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0933ede-8054-466c-82a5-83ad1751743b-multus-daemon-config\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-sysconfig\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.325462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-systemd\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325405    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41858fe6-7f55-456d-b7f7-59470406c3cf-tmp\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-system-cni-dir\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325502    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-os-release\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-var-lib-cni-multus\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr95n\" (UniqueName: \"kubernetes.io/projected/e0933ede-8054-466c-82a5-83ad1751743b-kube-api-access-rr95n\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74690fa1-9808-4516-81d4-6428a970246d-host-slash\") pod \"iptables-alerter-f62fq\" (UID: \"74690fa1-9808-4516-81d4-6428a970246d\") " pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-sys\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-var-lib-kubelet\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-tuned\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-system-cni-dir\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325771    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-modprobe-d\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-var-lib-cni-bin\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65a0ecad-5ba4-47c7-879d-3fd9aae552fa-konnectivity-ca\") pod \"konnectivity-agent-lffsb\" (UID: \"65a0ecad-5ba4-47c7-879d-3fd9aae552fa\") " pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:16.326123 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.325866    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tj6g\" (UniqueName: \"kubernetes.io/projected/74690fa1-9808-4516-81d4-6428a970246d-kube-api-access-8tj6g\") pod \"iptables-alerter-f62fq\" (UID: \"74690fa1-9808-4516-81d4-6428a970246d\") " pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.326925 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.322385    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.327813 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.327793    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-878h4\""
Jul 07 14:09:16.329681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.329483    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 14:09:16.329681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.329513    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 14:09:16.329952 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.329918    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 14:09:16.330094 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.330072    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 14:09:16.330712 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.330537    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bc72n\""
Jul 07 14:09:16.330712 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.330654    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.330879 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.330720    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 14:09:16.331067 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.331045    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 14:09:16.331785 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.331763    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:09:16.332893 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.332873    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 14:09:16.333072 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.333052    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 14:09:16.333175 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.333088    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mrvsb\""
Jul 07 14:09:16.352350 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.352318    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-db72x"
Jul 07 14:09:16.362807 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.362727    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-db72x"
Jul 07 14:09:16.413843 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.413744    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 14:09:16.426445 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-run-k8s-cni-cncf-io\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.426445 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-multus-conf-dir\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.426681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0933ede-8054-466c-82a5-83ad1751743b-multus-daemon-config\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.426681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426500    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-var-lib-openvswitch\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.426681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426567    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-multus-conf-dir\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.426681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426582    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-run-k8s-cni-cncf-io\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.426681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426623    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-cni-netd\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.426681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426647    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-sysconfig\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.426681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-systemd\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426739    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-systemd\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426766    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41858fe6-7f55-456d-b7f7-59470406c3cf-tmp\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426802    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-sysconfig\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-system-cni-dir\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426875    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-os-release\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426904    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-system-cni-dir\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426941    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr95n\" (UniqueName: \"kubernetes.io/projected/e0933ede-8054-466c-82a5-83ad1751743b-kube-api-access-rr95n\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.426968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52487f6a-4c05-47a6-881b-bc93718c61c8-env-overrides\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4963cb5e-680a-4bbf-8e68-258d0f5b7331-tmp-dir\") pod \"node-resolver-cfxrl\" (UID: \"4963cb5e-680a-4bbf-8e68-258d0f5b7331\") " pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427023    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-os-release\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.427044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-sys\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427156    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-tuned\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-system-cni-dir\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427202    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0933ede-8054-466c-82a5-83ad1751743b-multus-daemon-config\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-run-systemd\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427267    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-log-socket\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427292    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cab4996-d1b9-4425-ba8a-c5889b50acfb-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427292    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-system-cni-dir\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427317    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cab4996-d1b9-4425-ba8a-c5889b50acfb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.427577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427505    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-sys\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-run-multus-certs\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-device-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427739    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70208e8b-d198-4a27-abb8-248b9a6b9234-serviceca\") pod \"node-ca-n5pz8\" (UID: \"70208e8b-d198-4a27-abb8-248b9a6b9234\") " pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427766    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7zn7\" (UniqueName: \"kubernetes.io/projected/70208e8b-d198-4a27-abb8-248b9a6b9234-kube-api-access-k7zn7\") pod \"node-ca-n5pz8\" (UID: \"70208e8b-d198-4a27-abb8-248b9a6b9234\") " pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-multus-socket-dir-parent\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427815    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5f534d4-5cbf-4bbe-af34-41916ed91208-kubelet-config\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427830    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cab4996-d1b9-4425-ba8a-c5889b50acfb-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5f534d4-5cbf-4bbe-af34-41916ed91208-dbus\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-sysctl-d\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427889    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cab4996-d1b9-4425-ba8a-c5889b50acfb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427899    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-run-multus-certs\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427902    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-multus-cni-dir\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-etc-kubernetes\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.428045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427957    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-socket-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.427985    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52487f6a-4c05-47a6-881b-bc93718c61c8-ovnkube-script-lib\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-sysctl-conf\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428067    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-registration-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-etc-selinux\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428137    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-sysctl-d\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428213    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-multus-socket-dir-parent\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428260    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-etc-kubernetes\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428297    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428012    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-multus-cni-dir\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-run-netns\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428392    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-sysctl-conf\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.428696 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428400    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74690fa1-9808-4516-81d4-6428a970246d-host-slash\") pod \"iptables-alerter-f62fq\" (UID: \"74690fa1-9808-4516-81d4-6428a970246d\") " pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428427    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-run-netns\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-run-ovn\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428503    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74690fa1-9808-4516-81d4-6428a970246d-host-slash\") pod \"iptables-alerter-f62fq\" (UID: \"74690fa1-9808-4516-81d4-6428a970246d\") " pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428520    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52487f6a-4c05-47a6-881b-bc93718c61c8-ovnkube-config\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428525    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70208e8b-d198-4a27-abb8-248b9a6b9234-serviceca\") pod \"node-ca-n5pz8\" (UID: \"70208e8b-d198-4a27-abb8-248b9a6b9234\") " pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428542    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-run-netns\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-var-lib-cni-multus\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-etc-openvswitch\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428585    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-var-lib-cni-multus\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428608    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkbwh\" (UniqueName: \"kubernetes.io/projected/52487f6a-4c05-47a6-881b-bc93718c61c8-kube-api-access-kkbwh\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-var-lib-kubelet\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.429351 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-kubelet\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52487f6a-4c05-47a6-881b-bc93718c61c8-ovn-node-metrics-cert\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428709    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4963cb5e-680a-4bbf-8e68-258d0f5b7331-hosts-file\") pod \"node-resolver-cfxrl\" (UID: \"4963cb5e-680a-4bbf-8e68-258d0f5b7331\") " pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcnfs\" (UniqueName: \"kubernetes.io/projected/4963cb5e-680a-4bbf-8e68-258d0f5b7331-kube-api-access-pcnfs\") pod \"node-resolver-cfxrl\" (UID: \"4963cb5e-680a-4bbf-8e68-258d0f5b7331\") " pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428757    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-modprobe-d\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-var-lib-cni-bin\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428808    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65a0ecad-5ba4-47c7-879d-3fd9aae552fa-konnectivity-ca\") pod \"konnectivity-agent-lffsb\" (UID: \"65a0ecad-5ba4-47c7-879d-3fd9aae552fa\") " pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tj6g\" (UniqueName: \"kubernetes.io/projected/74690fa1-9808-4516-81d4-6428a970246d-kube-api-access-8tj6g\") pod \"iptables-alerter-f62fq\" (UID: \"74690fa1-9808-4516-81d4-6428a970246d\") " pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428860    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-systemd-units\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428885    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-run-openvswitch\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428912    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.428948    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.429035    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429073    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-var-lib-kubelet\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.430025 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.429142    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs podName:4298379e-f6c7-44eb-b922-f393e007c4af nodeName:}" failed. No retries permitted until 2026-07-07 14:09:16.929086267 +0000 UTC m=+3.037397861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs") pod "network-metrics-daemon-8jm6t" (UID: "4298379e-f6c7-44eb-b922-f393e007c4af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-run\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429189    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-modprobe-d\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-lib-modules\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429259    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-run\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429265    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-var-lib-cni-bin\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-var-lib-kubelet\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429317    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-hostroot\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429350    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65a0ecad-5ba4-47c7-879d-3fd9aae552fa-agent-certs\") pod \"konnectivity-agent-lffsb\" (UID: \"65a0ecad-5ba4-47c7-879d-3fd9aae552fa\") " pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429358    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-host-var-lib-kubelet\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8kq\" (UniqueName: \"kubernetes.io/projected/f363be3b-050c-4960-bec5-5e65de7f1098-kube-api-access-qm8kq\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429413    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-slash\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429437    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-node-log\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429450    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-lib-modules\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-kubernetes\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.430701 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429494    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-cnibin\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-cnibin\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429534    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-kubernetes\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-os-release\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429583    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-hostroot\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429591    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-cnibin\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429641    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cab4996-d1b9-4425-ba8a-c5889b50acfb-cnibin\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429648    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0933ede-8054-466c-82a5-83ad1751743b-os-release\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7bp7\" (UniqueName: \"kubernetes.io/projected/4298379e-f6c7-44eb-b922-f393e007c4af-kube-api-access-h7bp7\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429712    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429774    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65a0ecad-5ba4-47c7-879d-3fd9aae552fa-konnectivity-ca\") pod \"konnectivity-agent-lffsb\" (UID: \"65a0ecad-5ba4-47c7-879d-3fd9aae552fa\") " pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429825    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/74690fa1-9808-4516-81d4-6428a970246d-iptables-alerter-script\") pod \"iptables-alerter-f62fq\" (UID: \"74690fa1-9808-4516-81d4-6428a970246d\") " pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-cni-bin\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.429983    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-host\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.430025    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70208e8b-d198-4a27-abb8-248b9a6b9234-host\") pod \"node-ca-n5pz8\" (UID: \"70208e8b-d198-4a27-abb8-248b9a6b9234\") " pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.431482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.430086    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41858fe6-7f55-456d-b7f7-59470406c3cf-host\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.430119    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70208e8b-d198-4a27-abb8-248b9a6b9234-host\") pod \"node-ca-n5pz8\" (UID: \"70208e8b-d198-4a27-abb8-248b9a6b9234\") " pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.430727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cab4996-d1b9-4425-ba8a-c5889b50acfb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.430769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-sys-fs\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.430802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfzx5\" (UniqueName: \"kubernetes.io/projected/41858fe6-7f55-456d-b7f7-59470406c3cf-kube-api-access-wfzx5\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.430827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4xpg\" (UniqueName: \"kubernetes.io/projected/6cab4996-d1b9-4425-ba8a-c5889b50acfb-kube-api-access-k4xpg\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.430853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0933ede-8054-466c-82a5-83ad1751743b-cni-binary-copy\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.431057    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/74690fa1-9808-4516-81d4-6428a970246d-iptables-alerter-script\") pod \"iptables-alerter-f62fq\" (UID: \"74690fa1-9808-4516-81d4-6428a970246d\") " pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.431352    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0933ede-8054-466c-82a5-83ad1751743b-cni-binary-copy\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.431534    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cab4996-d1b9-4425-ba8a-c5889b50acfb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.432227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.431758    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41858fe6-7f55-456d-b7f7-59470406c3cf-etc-tuned\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.432691 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.432521    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41858fe6-7f55-456d-b7f7-59470406c3cf-tmp\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.432737 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.432690    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65a0ecad-5ba4-47c7-879d-3fd9aae552fa-agent-certs\") pod \"konnectivity-agent-lffsb\" (UID: \"65a0ecad-5ba4-47c7-879d-3fd9aae552fa\") " pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:16.437337 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.436851    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr95n\" (UniqueName: \"kubernetes.io/projected/e0933ede-8054-466c-82a5-83ad1751743b-kube-api-access-rr95n\") pod \"multus-785hx\" (UID: \"e0933ede-8054-466c-82a5-83ad1751743b\") " pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.437337 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.437194    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:16.437337 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.437215    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:16.437337 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.437228    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fdg86 for pod openshift-network-diagnostics/network-check-target-9r2vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:16.437337 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.437309    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86 podName:2ea869a2-8efc-4180-8f6a-5e4808dd6c28 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:16.937289693 +0000 UTC m=+3.045601280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fdg86" (UniqueName: "kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86") pod "network-check-target-9r2vh" (UID: "2ea869a2-8efc-4180-8f6a-5e4808dd6c28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:16.437848 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.437799    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tj6g\" (UniqueName: \"kubernetes.io/projected/74690fa1-9808-4516-81d4-6428a970246d-kube-api-access-8tj6g\") pod \"iptables-alerter-f62fq\" (UID: \"74690fa1-9808-4516-81d4-6428a970246d\") " pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.439685 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.439656    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7zn7\" (UniqueName: \"kubernetes.io/projected/70208e8b-d198-4a27-abb8-248b9a6b9234-kube-api-access-k7zn7\") pod \"node-ca-n5pz8\" (UID: \"70208e8b-d198-4a27-abb8-248b9a6b9234\") " pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.441166 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.441101    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfzx5\" (UniqueName: \"kubernetes.io/projected/41858fe6-7f55-456d-b7f7-59470406c3cf-kube-api-access-wfzx5\") pod \"tuned-t6zkp\" (UID: \"41858fe6-7f55-456d-b7f7-59470406c3cf\") " pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.441774 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.441749    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4xpg\" (UniqueName: \"kubernetes.io/projected/6cab4996-d1b9-4425-ba8a-c5889b50acfb-kube-api-access-k4xpg\") pod \"multus-additional-cni-plugins-4mkfq\" (UID: \"6cab4996-d1b9-4425-ba8a-c5889b50acfb\") " pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.442410 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.442360    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal" event={"ID":"0a2d38a3dcfd4cfda671d94785cdebdd","Type":"ContainerStarted","Data":"56ee231e74c608d895992947889c6c89ac36b647e128bb797a4c12cbf84936bc"}
Jul 07 14:09:16.443271 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.443224    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7bp7\" (UniqueName: \"kubernetes.io/projected/4298379e-f6c7-44eb-b922-f393e007c4af-kube-api-access-h7bp7\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:16.444003 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.443974    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal" event={"ID":"479338554c92fbd8bd392d8bacd28b2d","Type":"ContainerStarted","Data":"b492ff5d40bb9cdfe554d304a66642715543cf9a54aa2cfe55b6142763a50120"}
Jul 07 14:09:16.512979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.512943    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:16.531884 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.531837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkbwh\" (UniqueName: \"kubernetes.io/projected/52487f6a-4c05-47a6-881b-bc93718c61c8-kube-api-access-kkbwh\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.531884 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.531889    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-kubelet\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.531913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52487f6a-4c05-47a6-881b-bc93718c61c8-ovn-node-metrics-cert\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.531937    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4963cb5e-680a-4bbf-8e68-258d0f5b7331-hosts-file\") pod \"node-resolver-cfxrl\" (UID: \"4963cb5e-680a-4bbf-8e68-258d0f5b7331\") " pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.531964    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcnfs\" (UniqueName: \"kubernetes.io/projected/4963cb5e-680a-4bbf-8e68-258d0f5b7331-kube-api-access-pcnfs\") pod \"node-resolver-cfxrl\" (UID: \"4963cb5e-680a-4bbf-8e68-258d0f5b7331\") " pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.531967    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-kubelet\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.531991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-systemd-units\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532014    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-run-openvswitch\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532051    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532059    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4963cb5e-680a-4bbf-8e68-258d0f5b7331-hosts-file\") pod \"node-resolver-cfxrl\" (UID: \"4963cb5e-680a-4bbf-8e68-258d0f5b7331\") " pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8kq\" (UniqueName: \"kubernetes.io/projected/f363be3b-050c-4960-bec5-5e65de7f1098-kube-api-access-qm8kq\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-slash\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532143    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-node-log\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532179    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-cni-bin\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532220    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-sys-fs\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-var-lib-openvswitch\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-cni-netd\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52487f6a-4c05-47a6-881b-bc93718c61c8-env-overrides\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532327    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-run-openvswitch\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532332    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4963cb5e-680a-4bbf-8e68-258d0f5b7331-tmp-dir\") pod \"node-resolver-cfxrl\" (UID: \"4963cb5e-680a-4bbf-8e68-258d0f5b7331\") " pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532386    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-run-systemd\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532395    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-cni-bin\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-log-socket\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532442    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-slash\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532445    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-device-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532453    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-sys-fs\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532462    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-run-systemd\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.532614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532480    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5f534d4-5cbf-4bbe-af34-41916ed91208-kubelet-config\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532486    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-node-log\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532142    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-systemd-units\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532505    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5f534d4-5cbf-4bbe-af34-41916ed91208-dbus\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532520    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-var-lib-openvswitch\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-socket-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532539    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532545    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5f534d4-5cbf-4bbe-af34-41916ed91208-kubelet-config\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532557    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-log-socket\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532588    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-device-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532591    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52487f6a-4c05-47a6-881b-bc93718c61c8-ovnkube-script-lib\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532621    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4963cb5e-680a-4bbf-8e68-258d0f5b7331-tmp-dir\") pod \"node-resolver-cfxrl\" (UID: \"4963cb5e-680a-4bbf-8e68-258d0f5b7331\") " pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532623    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-cni-netd\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532649    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-socket-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.533216 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.532727    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532748    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5f534d4-5cbf-4bbe-af34-41916ed91208-dbus\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532777    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-registration-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.532822    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret podName:b5f534d4-5cbf-4bbe-af34-41916ed91208 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:17.032798379 +0000 UTC m=+3.141109964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret") pod "global-pull-secret-syncer-zhwzv" (UID: "b5f534d4-5cbf-4bbe-af34-41916ed91208") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532833    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-registration-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-etc-selinux\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-run-netns\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-run-ovn\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532944    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-etc-selinux\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532945    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532977    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532988    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52487f6a-4c05-47a6-881b-bc93718c61c8-ovnkube-config\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.533973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.533014    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-host-run-netns\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.534479 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.533017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-etc-openvswitch\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.534479 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.533023    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-run-ovn\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.534479 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.532984    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f363be3b-050c-4960-bec5-5e65de7f1098-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.534479 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.533044    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52487f6a-4c05-47a6-881b-bc93718c61c8-env-overrides\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.534479 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.533098    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52487f6a-4c05-47a6-881b-bc93718c61c8-etc-openvswitch\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.534479 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.533180    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52487f6a-4c05-47a6-881b-bc93718c61c8-ovnkube-script-lib\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.534479 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.534106    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52487f6a-4c05-47a6-881b-bc93718c61c8-ovnkube-config\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.535412 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.535382    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52487f6a-4c05-47a6-881b-bc93718c61c8-ovn-node-metrics-cert\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.542361 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.542288    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8kq\" (UniqueName: \"kubernetes.io/projected/f363be3b-050c-4960-bec5-5e65de7f1098-kube-api-access-qm8kq\") pod \"aws-ebs-csi-driver-node-gx988\" (UID: \"f363be3b-050c-4960-bec5-5e65de7f1098\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.543307 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.543280    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcnfs\" (UniqueName: \"kubernetes.io/projected/4963cb5e-680a-4bbf-8e68-258d0f5b7331-kube-api-access-pcnfs\") pod \"node-resolver-cfxrl\" (UID: \"4963cb5e-680a-4bbf-8e68-258d0f5b7331\") " pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.543866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.543842    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkbwh\" (UniqueName: \"kubernetes.io/projected/52487f6a-4c05-47a6-881b-bc93718c61c8-kube-api-access-kkbwh\") pod \"ovnkube-node-xzqcm\" (UID: \"52487f6a-4c05-47a6-881b-bc93718c61c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.622238 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.622062    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t6zkp"
Jul 07 14:09:16.636530 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.636495    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:16.644468 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.644435    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5pz8"
Jul 07 14:09:16.652418 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.652389    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4mkfq"
Jul 07 14:09:16.660297 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.660269    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f62fq"
Jul 07 14:09:16.669882 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.669842    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-785hx"
Jul 07 14:09:16.675807 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.675781    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988"
Jul 07 14:09:16.683845 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.683815    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:16.690833 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.690803    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cfxrl"
Jul 07 14:09:16.935269 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:16.935186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:16.935447 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.935308    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:16.935447 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:16.935365    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs podName:4298379e-f6c7-44eb-b922-f393e007c4af nodeName:}" failed. No retries permitted until 2026-07-07 14:09:17.935349444 +0000 UTC m=+4.043661021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs") pod "network-metrics-daemon-8jm6t" (UID: "4298379e-f6c7-44eb-b922-f393e007c4af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:17.036521 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.036488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:17.036691 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.036544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:17.036691 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:17.036665    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:17.036803 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:17.036747    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret podName:b5f534d4-5cbf-4bbe-af34-41916ed91208 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:18.036726339 +0000 UTC m=+4.145037925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret") pod "global-pull-secret-syncer-zhwzv" (UID: "b5f534d4-5cbf-4bbe-af34-41916ed91208") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:17.036803 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:17.036670    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:17.036803 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:17.036772    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:17.036803 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:17.036784    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fdg86 for pod openshift-network-diagnostics/network-check-target-9r2vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:17.036998 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:17.036829    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86 podName:2ea869a2-8efc-4180-8f6a-5e4808dd6c28 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:18.036817035 +0000 UTC m=+4.145128607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdg86" (UniqueName: "kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86") pod "network-check-target-9r2vh" (UID: "2ea869a2-8efc-4180-8f6a-5e4808dd6c28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:17.366997 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.366913    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:04:16 +0000 UTC" deadline="2027-12-08 16:39:02.114105613 +0000 UTC"
Jul 07 14:09:17.366997 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.366946    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12458h29m44.747162409s"
Jul 07 14:09:17.409666 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:17.409620    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0933ede_8054_466c_82a5_83ad1751743b.slice/crio-5aaf4a7a74e5ee04da6b3ed187d07fd6012c97a9ec844afa7fb04b129c0d469b WatchSource:0}: Error finding container 5aaf4a7a74e5ee04da6b3ed187d07fd6012c97a9ec844afa7fb04b129c0d469b: Status 404 returned error can't find the container with id 5aaf4a7a74e5ee04da6b3ed187d07fd6012c97a9ec844afa7fb04b129c0d469b
Jul 07 14:09:17.411871 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:17.411841    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf363be3b_050c_4960_bec5_5e65de7f1098.slice/crio-b9daccea41c450a8d68bc19e6819902ccd2edf0681b62c02df4dc2962ec6960b WatchSource:0}: Error finding container b9daccea41c450a8d68bc19e6819902ccd2edf0681b62c02df4dc2962ec6960b: Status 404 returned error can't find the container with id b9daccea41c450a8d68bc19e6819902ccd2edf0681b62c02df4dc2962ec6960b
Jul 07 14:09:17.415029 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:17.414969    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52487f6a_4c05_47a6_881b_bc93718c61c8.slice/crio-5be2b1ded6b348eaf38f653871e472379004622e313ed03151bbe2d3f1d509b0 WatchSource:0}: Error finding container 5be2b1ded6b348eaf38f653871e472379004622e313ed03151bbe2d3f1d509b0: Status 404 returned error can't find the container with id 5be2b1ded6b348eaf38f653871e472379004622e313ed03151bbe2d3f1d509b0
Jul 07 14:09:17.416446 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:17.416417    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74690fa1_9808_4516_81d4_6428a970246d.slice/crio-cd1e981e1da5c766f3431983685254bb5250784128d12f27d3544c53c28a5607 WatchSource:0}: Error finding container cd1e981e1da5c766f3431983685254bb5250784128d12f27d3544c53c28a5607: Status 404 returned error can't find the container with id cd1e981e1da5c766f3431983685254bb5250784128d12f27d3544c53c28a5607
Jul 07 14:09:17.417246 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:17.417175    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a0ecad_5ba4_47c7_879d_3fd9aae552fa.slice/crio-039fc75d8c125dcc4a5d04d1442e859815cfd2d2e50f44481954ac94bc49f01a WatchSource:0}: Error finding container 039fc75d8c125dcc4a5d04d1442e859815cfd2d2e50f44481954ac94bc49f01a: Status 404 returned error can't find the container with id 039fc75d8c125dcc4a5d04d1442e859815cfd2d2e50f44481954ac94bc49f01a
Jul 07 14:09:17.419403 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:17.419085    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70208e8b_d198_4a27_abb8_248b9a6b9234.slice/crio-d0accd317b936be79ef2bce5d998206cff7fd1ac992922da6bbd0acdc2f697fe WatchSource:0}: Error finding container d0accd317b936be79ef2bce5d998206cff7fd1ac992922da6bbd0acdc2f697fe: Status 404 returned error can't find the container with id d0accd317b936be79ef2bce5d998206cff7fd1ac992922da6bbd0acdc2f697fe
Jul 07 14:09:17.420892 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:17.420868    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cab4996_d1b9_4425_ba8a_c5889b50acfb.slice/crio-93bf657e85bf6b765ef4ef6608975be040b742320acf6c40460576263937b4b8 WatchSource:0}: Error finding container 93bf657e85bf6b765ef4ef6608975be040b742320acf6c40460576263937b4b8: Status 404 returned error can't find the container with id 93bf657e85bf6b765ef4ef6608975be040b742320acf6c40460576263937b4b8
Jul 07 14:09:17.421543 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:17.421519    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4963cb5e_680a_4bbf_8e68_258d0f5b7331.slice/crio-980fbb4aee46556bad382ad1bb45b60cc0b9a91f748ae7d48a4e22944449c519 WatchSource:0}: Error finding container 980fbb4aee46556bad382ad1bb45b60cc0b9a91f748ae7d48a4e22944449c519: Status 404 returned error can't find the container with id 980fbb4aee46556bad382ad1bb45b60cc0b9a91f748ae7d48a4e22944449c519
Jul 07 14:09:17.423607 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:17.423576    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41858fe6_7f55_456d_b7f7_59470406c3cf.slice/crio-bb6aa68b713e6325744be7e50563e74dc49df5499ae4cd20cef804310c96de16 WatchSource:0}: Error finding container bb6aa68b713e6325744be7e50563e74dc49df5499ae4cd20cef804310c96de16: Status 404 returned error can't find the container with id bb6aa68b713e6325744be7e50563e74dc49df5499ae4cd20cef804310c96de16
Jul 07 14:09:17.446686 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.446651    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mkfq" event={"ID":"6cab4996-d1b9-4425-ba8a-c5889b50acfb","Type":"ContainerStarted","Data":"93bf657e85bf6b765ef4ef6608975be040b742320acf6c40460576263937b4b8"}
Jul 07 14:09:17.447715 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.447687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t6zkp" event={"ID":"41858fe6-7f55-456d-b7f7-59470406c3cf","Type":"ContainerStarted","Data":"bb6aa68b713e6325744be7e50563e74dc49df5499ae4cd20cef804310c96de16"}
Jul 07 14:09:17.448799 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.448767    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cfxrl" event={"ID":"4963cb5e-680a-4bbf-8e68-258d0f5b7331","Type":"ContainerStarted","Data":"980fbb4aee46556bad382ad1bb45b60cc0b9a91f748ae7d48a4e22944449c519"}
Jul 07 14:09:17.449891 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.449867    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f62fq" event={"ID":"74690fa1-9808-4516-81d4-6428a970246d","Type":"ContainerStarted","Data":"cd1e981e1da5c766f3431983685254bb5250784128d12f27d3544c53c28a5607"}
Jul 07 14:09:17.451127 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.451084    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" event={"ID":"52487f6a-4c05-47a6-881b-bc93718c61c8","Type":"ContainerStarted","Data":"5be2b1ded6b348eaf38f653871e472379004622e313ed03151bbe2d3f1d509b0"}
Jul 07 14:09:17.452226 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.452201    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988" event={"ID":"f363be3b-050c-4960-bec5-5e65de7f1098","Type":"ContainerStarted","Data":"b9daccea41c450a8d68bc19e6819902ccd2edf0681b62c02df4dc2962ec6960b"}
Jul 07 14:09:17.453471 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.453286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5pz8" event={"ID":"70208e8b-d198-4a27-abb8-248b9a6b9234","Type":"ContainerStarted","Data":"d0accd317b936be79ef2bce5d998206cff7fd1ac992922da6bbd0acdc2f697fe"}
Jul 07 14:09:17.454379 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.454352    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lffsb" event={"ID":"65a0ecad-5ba4-47c7-879d-3fd9aae552fa","Type":"ContainerStarted","Data":"039fc75d8c125dcc4a5d04d1442e859815cfd2d2e50f44481954ac94bc49f01a"}
Jul 07 14:09:17.456758 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.456729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-785hx" event={"ID":"e0933ede-8054-466c-82a5-83ad1751743b","Type":"ContainerStarted","Data":"5aaf4a7a74e5ee04da6b3ed187d07fd6012c97a9ec844afa7fb04b129c0d469b"}
Jul 07 14:09:17.942419 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:17.942329    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:17.942877 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:17.942583    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:17.942877 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:17.942653    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs podName:4298379e-f6c7-44eb-b922-f393e007c4af nodeName:}" failed. No retries permitted until 2026-07-07 14:09:19.942634343 +0000 UTC m=+6.050945917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs") pod "network-metrics-daemon-8jm6t" (UID: "4298379e-f6c7-44eb-b922-f393e007c4af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:18.043504 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.043461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:18.043694 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.043547    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:18.043764 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:18.043706    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:18.043813 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:18.043770    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret podName:b5f534d4-5cbf-4bbe-af34-41916ed91208 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:20.043750612 +0000 UTC m=+6.152062188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret") pod "global-pull-secret-syncer-zhwzv" (UID: "b5f534d4-5cbf-4bbe-af34-41916ed91208") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:18.044231 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:18.044211    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:18.044318 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:18.044237    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:18.044318 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:18.044252    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fdg86 for pod openshift-network-diagnostics/network-check-target-9r2vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:18.044318 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:18.044300    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86 podName:2ea869a2-8efc-4180-8f6a-5e4808dd6c28 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:20.04428556 +0000 UTC m=+6.152597136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdg86" (UniqueName: "kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86") pod "network-check-target-9r2vh" (UID: "2ea869a2-8efc-4180-8f6a-5e4808dd6c28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:18.368139 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.368000    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:04:16 +0000 UTC" deadline="2028-02-03 01:25:24.253119625 +0000 UTC"
Jul 07 14:09:18.368139 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.368071    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13811h16m5.885053552s"
Jul 07 14:09:18.437279 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.437201    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:18.437463 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:18.437344    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:18.437851 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.437587    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:18.437851 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.437659    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:18.437851 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:18.437749    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:18.437851 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:18.437823    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:18.469314 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.469272    2569 generic.go:356] "Generic (PLEG): container finished" podID="0a2d38a3dcfd4cfda671d94785cdebdd" containerID="127a2ffb409c670699a6dd5a50ccfd499ebf53612b8c2264f7730c9a5f592c44" exitCode=0
Jul 07 14:09:18.469482 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.469393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal" event={"ID":"0a2d38a3dcfd4cfda671d94785cdebdd","Type":"ContainerDied","Data":"127a2ffb409c670699a6dd5a50ccfd499ebf53612b8c2264f7730c9a5f592c44"}
Jul 07 14:09:18.485740 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.485652    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal" event={"ID":"479338554c92fbd8bd392d8bacd28b2d","Type":"ContainerStarted","Data":"be7b9135adc051f14d28fb9dd6e6f923de404545654c2b8cfd41a8d93fd90e60"}
Jul 07 14:09:18.503495 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:18.503425    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-29.ec2.internal" podStartSLOduration=3.503401085 podStartE2EDuration="3.503401085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:18.503182854 +0000 UTC m=+4.611494459" watchObservedRunningTime="2026-07-07 14:09:18.503401085 +0000 UTC m=+4.611712679"
Jul 07 14:09:19.495639 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:19.495602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal" event={"ID":"0a2d38a3dcfd4cfda671d94785cdebdd","Type":"ContainerStarted","Data":"b63d4d21d731d9cce5dfb59565150e0b46f01376b231ad35a2d136983cf289a7"}
Jul 07 14:09:19.960752 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:19.959938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:19.960752 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:19.960225    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:19.960752 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:19.960292    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs podName:4298379e-f6c7-44eb-b922-f393e007c4af nodeName:}" failed. No retries permitted until 2026-07-07 14:09:23.960271344 +0000 UTC m=+10.068582930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs") pod "network-metrics-daemon-8jm6t" (UID: "4298379e-f6c7-44eb-b922-f393e007c4af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:20.061806 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:20.060947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:20.061806 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:20.061031    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:20.061806 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:20.061212    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:20.061806 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:20.061280    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret podName:b5f534d4-5cbf-4bbe-af34-41916ed91208 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:24.06126191 +0000 UTC m=+10.169573488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret") pod "global-pull-secret-syncer-zhwzv" (UID: "b5f534d4-5cbf-4bbe-af34-41916ed91208") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:20.061806 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:20.061695    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:20.061806 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:20.061713    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:20.061806 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:20.061726    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fdg86 for pod openshift-network-diagnostics/network-check-target-9r2vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:20.061806 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:20.061768    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86 podName:2ea869a2-8efc-4180-8f6a-5e4808dd6c28 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:24.06175368 +0000 UTC m=+10.170065252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdg86" (UniqueName: "kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86") pod "network-check-target-9r2vh" (UID: "2ea869a2-8efc-4180-8f6a-5e4808dd6c28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:20.443735 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:20.443652    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:20.443906 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:20.443786    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:20.444240 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:20.444218    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:20.444334 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:20.444227    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:20.444398 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:20.444339    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:20.444454 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:20.444430    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:22.436410 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:22.435879    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:22.436410 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:22.436023    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:22.436410 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:22.436065    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:22.436410 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:22.436189    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:22.436410 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:22.436230    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:22.436410 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:22.436293    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:23.996683 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:23.996629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:23.997192 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:23.996788    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:23.997192 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:23.996875    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs podName:4298379e-f6c7-44eb-b922-f393e007c4af nodeName:}" failed. No retries permitted until 2026-07-07 14:09:31.996850465 +0000 UTC m=+18.105162041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs") pod "network-metrics-daemon-8jm6t" (UID: "4298379e-f6c7-44eb-b922-f393e007c4af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:24.097501 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:24.097458    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:24.097688 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:24.097544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:24.097688 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:24.097680    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:24.097808 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:24.097737    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret podName:b5f534d4-5cbf-4bbe-af34-41916ed91208 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:32.09772093 +0000 UTC m=+18.206032502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret") pod "global-pull-secret-syncer-zhwzv" (UID: "b5f534d4-5cbf-4bbe-af34-41916ed91208") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:24.097867 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:24.097821    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:24.097867 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:24.097831    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:24.097867 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:24.097840    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fdg86 for pod openshift-network-diagnostics/network-check-target-9r2vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:24.098002 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:24.097871    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86 podName:2ea869a2-8efc-4180-8f6a-5e4808dd6c28 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:32.097858638 +0000 UTC m=+18.206170234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdg86" (UniqueName: "kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86") pod "network-check-target-9r2vh" (UID: "2ea869a2-8efc-4180-8f6a-5e4808dd6c28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:24.437293 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:24.436889    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:24.437293 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:24.436983    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:24.437293 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:24.437018    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:24.437293 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:24.437046    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:24.437293 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:24.437097    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:24.437293 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:24.437202    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:26.435673 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:26.435632    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:26.435673 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:26.435664    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:26.436158 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:26.435774    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:26.436158 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:26.435824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:26.436158 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:26.435929    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:26.436158 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:26.436038    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:28.435197 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:28.435165    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:28.435717 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:28.435167    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:28.435717 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:28.435306    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:28.435717 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:28.435410    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:28.435717 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:28.435167    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:28.435717 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:28.435552    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:30.435463 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:30.435416    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:30.435890 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:30.435475    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:30.435890 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:30.435429    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:30.435890 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:30.435565    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:30.435890 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:30.435680    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:30.435890 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:30.435753    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:32.056133 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:32.056085    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:32.056541 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.056252    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:32.056541 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.056315    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs podName:4298379e-f6c7-44eb-b922-f393e007c4af nodeName:}" failed. No retries permitted until 2026-07-07 14:09:48.056297502 +0000 UTC m=+34.164609086 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs") pod "network-metrics-daemon-8jm6t" (UID: "4298379e-f6c7-44eb-b922-f393e007c4af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:32.156863 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:32.156821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:32.157055 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:32.156904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:32.157055 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.156965    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:32.157055 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.157031    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:32.157055 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.157047    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:32.157253 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.157062    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fdg86 for pod openshift-network-diagnostics/network-check-target-9r2vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:32.157253 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.157056    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret podName:b5f534d4-5cbf-4bbe-af34-41916ed91208 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:48.157034606 +0000 UTC m=+34.265346202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret") pod "global-pull-secret-syncer-zhwzv" (UID: "b5f534d4-5cbf-4bbe-af34-41916ed91208") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:32.157253 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.157133    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86 podName:2ea869a2-8efc-4180-8f6a-5e4808dd6c28 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:48.15709986 +0000 UTC m=+34.265411448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdg86" (UniqueName: "kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86") pod "network-check-target-9r2vh" (UID: "2ea869a2-8efc-4180-8f6a-5e4808dd6c28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:32.435033 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:32.434943    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:32.435033 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:32.434988    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:32.435033 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:32.434989    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:32.435299 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.435075    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:32.435299 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.435182    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:32.435299 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:32.435245    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:34.436316 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:34.436275    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:34.436790 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:34.436322    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:34.436790 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:34.436400    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:34.436790 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:34.436456    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:34.436790 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:34.436496    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:34.436790 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:34.436579    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:36.435337 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.435057    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:36.436279 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.435053    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:36.436279 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.435104    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:36.436279 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:36.435478    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:36.436279 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:36.435571    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:36.436279 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:36.435676    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:36.529900 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.529864    2569 generic.go:356] "Generic (PLEG): container finished" podID="6cab4996-d1b9-4425-ba8a-c5889b50acfb" containerID="b92ec6ddab7b11e6cf196f0334367babbb9a4a95b8ee996bcf87585257aae107" exitCode=0
Jul 07 14:09:36.530086 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.529947    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mkfq" event={"ID":"6cab4996-d1b9-4425-ba8a-c5889b50acfb","Type":"ContainerDied","Data":"b92ec6ddab7b11e6cf196f0334367babbb9a4a95b8ee996bcf87585257aae107"}
Jul 07 14:09:36.531528 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.531500    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t6zkp" event={"ID":"41858fe6-7f55-456d-b7f7-59470406c3cf","Type":"ContainerStarted","Data":"9b4c91dc7b141681cf7a70a4dc5085b3cc984dd5ce12507d62096bca16da4d0a"}
Jul 07 14:09:36.532953 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.532921    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cfxrl" event={"ID":"4963cb5e-680a-4bbf-8e68-258d0f5b7331","Type":"ContainerStarted","Data":"720197fb79afd5eb86269c1ed1fec53ac88bef4d788ebf6cc77975128e08846b"}
Jul 07 14:09:36.535242 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.535210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" event={"ID":"52487f6a-4c05-47a6-881b-bc93718c61c8","Type":"ContainerStarted","Data":"33a79ca8abe9ca580f6cbc1831b5ae18b450f0af2f62cfd595b4c6d09a76309b"}
Jul 07 14:09:36.535376 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.535284    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" event={"ID":"52487f6a-4c05-47a6-881b-bc93718c61c8","Type":"ContainerStarted","Data":"0aacdcff4cf543e56868f7793283518ddf28cc01d0fca9f74ab78853ef2af8a1"}
Jul 07 14:09:36.535376 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.535331    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" event={"ID":"52487f6a-4c05-47a6-881b-bc93718c61c8","Type":"ContainerStarted","Data":"537c7264ea4c52a2ddcccf3cacb780dec6be0c0c7fc2540a47941875a6f16b7a"}
Jul 07 14:09:36.536749 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.536723    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988" event={"ID":"f363be3b-050c-4960-bec5-5e65de7f1098","Type":"ContainerStarted","Data":"1238a8212cfb705b9e62b8510fccfbac24af34f5d03c6d368b3deb6723d2c2e7"}
Jul 07 14:09:36.538290 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.538263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5pz8" event={"ID":"70208e8b-d198-4a27-abb8-248b9a6b9234","Type":"ContainerStarted","Data":"1f1d4ff9212237c70031d2198eb564e80864ff9ae44f79896a0c42e29768e79d"}
Jul 07 14:09:36.539656 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.539628    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lffsb" event={"ID":"65a0ecad-5ba4-47c7-879d-3fd9aae552fa","Type":"ContainerStarted","Data":"92f57248c8bc948aa41eee281f5b2d47a505d2978737560094bfccb674d449df"}
Jul 07 14:09:36.541124 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.541086    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-785hx" event={"ID":"e0933ede-8054-466c-82a5-83ad1751743b","Type":"ContainerStarted","Data":"7e775706997cd0a639617d5df6e171358047c0d655ee4f8b5314080db05112bc"}
Jul 07 14:09:36.554260 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.554203    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-29.ec2.internal" podStartSLOduration=21.554185152 podStartE2EDuration="21.554185152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:19.513656716 +0000 UTC m=+5.621968311" watchObservedRunningTime="2026-07-07 14:09:36.554185152 +0000 UTC m=+22.662496745"
Jul 07 14:09:36.571124 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.571067    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t6zkp" podStartSLOduration=3.9196308479999997 podStartE2EDuration="22.571051096s" totalImagesPullingTime="18.651420248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:36.57077614 +0000 UTC m=+22.679087747" watchObservedRunningTime="2026-07-07 14:09:36.571051096 +0000 UTC m=+22.679362690"
Jul 07 14:09:36.588104 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.588049    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-785hx" podStartSLOduration=3.867228408 podStartE2EDuration="22.588032802s" totalImagesPullingTime="18.720804394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:36.587695807 +0000 UTC m=+22.696007400" watchObservedRunningTime="2026-07-07 14:09:36.588032802 +0000 UTC m=+22.696344395"
Jul 07 14:09:36.631589 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.631539    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lffsb" podStartSLOduration=12.304811187 podStartE2EDuration="22.631520203s" totalImagesPullingTime="10.326709016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:36.631105507 +0000 UTC m=+22.739417097" watchObservedRunningTime="2026-07-07 14:09:36.631520203 +0000 UTC m=+22.739831818"
Jul 07 14:09:36.652322 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.652269    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n5pz8" podStartSLOduration=4.001870644 podStartE2EDuration="22.652250904s" totalImagesPullingTime="18.65038026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:36.652172127 +0000 UTC m=+22.760483722" watchObservedRunningTime="2026-07-07 14:09:36.652250904 +0000 UTC m=+22.760562497"
Jul 07 14:09:36.670235 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:36.670186    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cfxrl" podStartSLOduration=4.022383041 podStartE2EDuration="22.670167939s" totalImagesPullingTime="18.647784898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:36.669839801 +0000 UTC m=+22.778151395" watchObservedRunningTime="2026-07-07 14:09:36.670167939 +0000 UTC m=+22.778479533"
Jul 07 14:09:37.547851 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:37.547797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" event={"ID":"52487f6a-4c05-47a6-881b-bc93718c61c8","Type":"ContainerStarted","Data":"cb9e6f4d10d8216170c4d65656e9d67b8f95e02c3a8dd00e56f503f52e5abc50"}
Jul 07 14:09:37.548330 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:37.547962    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" event={"ID":"52487f6a-4c05-47a6-881b-bc93718c61c8","Type":"ContainerStarted","Data":"4d56b36338448a794a8e883710a4d97df331c82c868a8ae56260e14701628a61"}
Jul 07 14:09:37.548330 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:37.548021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" event={"ID":"52487f6a-4c05-47a6-881b-bc93718c61c8","Type":"ContainerStarted","Data":"579950a1206e53cd0df41486acae76e0a81bf35cd8387186c9e43a2c506a55c6"}
Jul 07 14:09:38.291171 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.291127    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 14:09:38.379059 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.378967    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:38.379612 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.379590    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:38.401425 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.401285    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:09:38.29115435Z","UUID":"b965f439-e5e3-4564-ac7a-41d43e83bb3b","Handler":null,"Name":"","Endpoint":""}
Jul 07 14:09:38.403025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.402992    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 14:09:38.403185 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.403038    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 14:09:38.435128 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.435084    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:38.435128 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.435125    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:38.435353 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.435084    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:38.435353 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:38.435228    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:38.435353 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:38.435270    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:38.435353 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:38.435340    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:38.552083 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.551834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f62fq" event={"ID":"74690fa1-9808-4516-81d4-6428a970246d","Type":"ContainerStarted","Data":"d5eafe16f6c09783840453cb233facbe0fedc9b31376ec1b68f6fe41cfd275e3"}
Jul 07 14:09:38.554080 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.554020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988" event={"ID":"f363be3b-050c-4960-bec5-5e65de7f1098","Type":"ContainerStarted","Data":"bcf10eeea0fc88343497461b8129424457b99b5669f79a5dc63d7c5ef9f5a5dc"}
Jul 07 14:09:38.554355 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.554329    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:38.554870 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.554846    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lffsb"
Jul 07 14:09:38.569068 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:38.569003    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f62fq" podStartSLOduration=5.916259789 podStartE2EDuration="24.568981188s" totalImagesPullingTime="18.652721399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:38.568319937 +0000 UTC m=+24.676631533" watchObservedRunningTime="2026-07-07 14:09:38.568981188 +0000 UTC m=+24.677292784"
Jul 07 14:09:39.560185 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:39.560088    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" event={"ID":"52487f6a-4c05-47a6-881b-bc93718c61c8","Type":"ContainerStarted","Data":"44c6629417b035b0d4d476bcf4c097aa92de81382e6e2543c0492e8ab9cef4ad"}
Jul 07 14:09:40.435182 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:40.435141    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:40.435373 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:40.435150    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:40.435373 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:40.435279    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:40.435472 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:40.435378    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:40.435472 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:40.435150    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:40.435564 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:40.435476    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:41.566486 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:41.566276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988" event={"ID":"f363be3b-050c-4960-bec5-5e65de7f1098","Type":"ContainerStarted","Data":"0f992e884c8c84f235507d5b0f5d6482808a66ed42d0c19546c6d4d6c51d6a58"}
Jul 07 14:09:41.568088 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:41.568061    2569 generic.go:356] "Generic (PLEG): container finished" podID="6cab4996-d1b9-4425-ba8a-c5889b50acfb" containerID="ae8e3153b497da8b6aca3269dd829d8a690d450d12014035b114d9956e25c187" exitCode=0
Jul 07 14:09:41.568231 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:41.568148    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mkfq" event={"ID":"6cab4996-d1b9-4425-ba8a-c5889b50acfb","Type":"ContainerDied","Data":"ae8e3153b497da8b6aca3269dd829d8a690d450d12014035b114d9956e25c187"}
Jul 07 14:09:41.572230 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:41.572202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" event={"ID":"52487f6a-4c05-47a6-881b-bc93718c61c8","Type":"ContainerStarted","Data":"6c91d8bf40ee02dcd2273979349ef4170cb706f70ae16e52fc514575fdccc4a7"}
Jul 07 14:09:41.572469 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:41.572454    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:41.586378 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:41.586325    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx988" podStartSLOduration=4.077344606 podStartE2EDuration="27.586308369s" totalImagesPullingTime="23.508963763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:41.58601956 +0000 UTC m=+27.694331154" watchObservedRunningTime="2026-07-07 14:09:41.586308369 +0000 UTC m=+27.694619963"
Jul 07 14:09:41.589679 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:41.589649    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:41.643826 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:41.643773    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm" podStartSLOduration=8.936741658 podStartE2EDuration="27.643756271s" totalImagesPullingTime="18.707014613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:41.643525112 +0000 UTC m=+27.751836707" watchObservedRunningTime="2026-07-07 14:09:41.643756271 +0000 UTC m=+27.752067864"
Jul 07 14:09:42.435861 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:42.435824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:42.436061 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:42.435824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:42.436061 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:42.435943    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:42.436061 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:42.436043    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:42.436240 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:42.435832    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:42.436240 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:42.436193    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:42.583623 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:42.580828    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:42.583623 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:42.580907    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:42.616150 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:42.616087    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:09:43.337886 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:43.337849    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zhwzv"]
Jul 07 14:09:43.338164 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:43.337951    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:43.338164 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:43.338035    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:43.343422 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:43.343393    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9r2vh"]
Jul 07 14:09:43.343627 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:43.343486    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:43.343627 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:43.343580    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:43.344381 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:43.344356    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8jm6t"]
Jul 07 14:09:43.344509 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:43.344449    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:43.344558 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:43.344534    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:43.583609 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:43.583402    2569 generic.go:356] "Generic (PLEG): container finished" podID="6cab4996-d1b9-4425-ba8a-c5889b50acfb" containerID="b09166f5656d9e395addf29b765bec6c5dcef5b2b03d253a3335bf670b71e965" exitCode=0
Jul 07 14:09:43.583780 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:43.583490    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mkfq" event={"ID":"6cab4996-d1b9-4425-ba8a-c5889b50acfb","Type":"ContainerDied","Data":"b09166f5656d9e395addf29b765bec6c5dcef5b2b03d253a3335bf670b71e965"}
Jul 07 14:09:44.435847 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:44.435812    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:44.436057 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:44.435926    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:45.435035 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:45.434993    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:45.435035 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:45.435024    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:45.435792 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:45.435136    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:45.435792 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:45.435180    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:45.590612 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:45.590580    2569 generic.go:356] "Generic (PLEG): container finished" podID="6cab4996-d1b9-4425-ba8a-c5889b50acfb" containerID="98f6a5b830f5aa54dfb6714dbd38e4818e9c332c22a85c23cdc96a4ce9f9e841" exitCode=0
Jul 07 14:09:45.590612 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:45.590623    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mkfq" event={"ID":"6cab4996-d1b9-4425-ba8a-c5889b50acfb","Type":"ContainerDied","Data":"98f6a5b830f5aa54dfb6714dbd38e4818e9c332c22a85c23cdc96a4ce9f9e841"}
Jul 07 14:09:46.435991 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:46.435959    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:46.436789 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:46.436086    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:47.435209 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:47.435168    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:47.435372 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:47.435175    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:47.435372 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:47.435317    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2vh" podUID="2ea869a2-8efc-4180-8f6a-5e4808dd6c28"
Jul 07 14:09:47.435488 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:47.435415    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jm6t" podUID="4298379e-f6c7-44eb-b922-f393e007c4af"
Jul 07 14:09:48.075819 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:48.075775    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:48.076304 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:48.075908    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:48.076304 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:48.075975    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs podName:4298379e-f6c7-44eb-b922-f393e007c4af nodeName:}" failed. No retries permitted until 2026-07-07 14:10:20.075960587 +0000 UTC m=+66.184272165 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs") pod "network-metrics-daemon-8jm6t" (UID: "4298379e-f6c7-44eb-b922-f393e007c4af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:48.177090 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:48.177050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:48.177299 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:48.177152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:48.177299 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:48.177223    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:48.177299 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:48.177272    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:48.177299 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:48.177287    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:48.177299 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:48.177294    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret podName:b5f534d4-5cbf-4bbe-af34-41916ed91208 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:20.177276395 +0000 UTC m=+66.285587973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret") pod "global-pull-secret-syncer-zhwzv" (UID: "b5f534d4-5cbf-4bbe-af34-41916ed91208") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:48.177299 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:48.177301    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fdg86 for pod openshift-network-diagnostics/network-check-target-9r2vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:48.177566 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:48.177335    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86 podName:2ea869a2-8efc-4180-8f6a-5e4808dd6c28 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:20.177325385 +0000 UTC m=+66.285636970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdg86" (UniqueName: "kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86") pod "network-check-target-9r2vh" (UID: "2ea869a2-8efc-4180-8f6a-5e4808dd6c28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:48.435265 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:48.435226    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:48.435467 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:48.435361    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhwzv" podUID="b5f534d4-5cbf-4bbe-af34-41916ed91208"
Jul 07 14:09:49.126141 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.126077    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-29.ec2.internal" event="NodeReady"
Jul 07 14:09:49.126609 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.126283    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 14:09:49.169537 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.169031    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f47d4797f-k8jfq"]
Jul 07 14:09:49.175499 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.175463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.178398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.178179    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 14:09:49.178398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.178207    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 14:09:49.178398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.178325    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 14:09:49.179176 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.179150    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2kvq8"]
Jul 07 14:09:49.179300 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.179271    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bstgj\""
Jul 07 14:09:49.183376 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.183241    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.184068 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.184029    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 14:09:49.185945 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.185918    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 14:09:49.186206 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.186191    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 14:09:49.186302 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.186285    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-njxhm\""
Jul 07 14:09:49.188938 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.188916    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f47d4797f-k8jfq"]
Jul 07 14:09:49.195512 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.195485    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9snb9"]
Jul 07 14:09:49.199693 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.199666    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2kvq8"]
Jul 07 14:09:49.199826 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.199802    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9snb9"
Jul 07 14:09:49.204127 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.204088    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 14:09:49.204417 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.204390    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2bzq9\""
Jul 07 14:09:49.204544 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.204419    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 14:09:49.204606 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.204556    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 14:09:49.212853 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.212758    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9snb9"]
Jul 07 14:09:49.285072 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-trusted-ca\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.285267 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-image-registry-private-configuration\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.285267 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285134    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-bound-sa-token\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.285267 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285157    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8npt\" (UniqueName: \"kubernetes.io/projected/27acbc3b-ba1e-411b-a893-a78af377f3ad-kube-api-access-j8npt\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.285267 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285178    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-registry-certificates\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.285267 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285199    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/27acbc3b-ba1e-411b-a893-a78af377f3ad-config-volume\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.285267 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.285267 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285244    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b91a946-afd8-446e-af80-613ed4826f31-ca-trust-extracted\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.285568 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285303    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27acbc3b-ba1e-411b-a893-a78af377f3ad-tmp-dir\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.285568 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285392    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.285568 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285423    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-installation-pull-secrets\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.285568 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcbv5\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-kube-api-access-wcbv5\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.285568 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285535    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2799z\" (UniqueName: \"kubernetes.io/projected/41251dd9-eb69-46b2-a2c3-fbabac0dc673-kube-api-access-2799z\") pod \"ingress-canary-9snb9\" (UID: \"41251dd9-eb69-46b2-a2c3-fbabac0dc673\") " pod="openshift-ingress-canary/ingress-canary-9snb9"
Jul 07 14:09:49.285750 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.285571    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41251dd9-eb69-46b2-a2c3-fbabac0dc673-cert\") pod \"ingress-canary-9snb9\" (UID: \"41251dd9-eb69-46b2-a2c3-fbabac0dc673\") " pod="openshift-ingress-canary/ingress-canary-9snb9"
Jul 07 14:09:49.386545 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-trusted-ca\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.386545 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-image-registry-private-configuration\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.386545 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386532    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-bound-sa-token\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.386822 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8npt\" (UniqueName: \"kubernetes.io/projected/27acbc3b-ba1e-411b-a893-a78af377f3ad-kube-api-access-j8npt\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.386822 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-registry-certificates\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.386822 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/27acbc3b-ba1e-411b-a893-a78af377f3ad-config-volume\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.386822 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.386822 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386731    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b91a946-afd8-446e-af80-613ed4826f31-ca-trust-extracted\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.386822 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27acbc3b-ba1e-411b-a893-a78af377f3ad-tmp-dir\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.386822 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.387175 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-installation-pull-secrets\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.387175 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcbv5\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-kube-api-access-wcbv5\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.387175 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2799z\" (UniqueName: \"kubernetes.io/projected/41251dd9-eb69-46b2-a2c3-fbabac0dc673-kube-api-access-2799z\") pod \"ingress-canary-9snb9\" (UID: \"41251dd9-eb69-46b2-a2c3-fbabac0dc673\") " pod="openshift-ingress-canary/ingress-canary-9snb9"
Jul 07 14:09:49.387175 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.386962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41251dd9-eb69-46b2-a2c3-fbabac0dc673-cert\") pod \"ingress-canary-9snb9\" (UID: \"41251dd9-eb69-46b2-a2c3-fbabac0dc673\") " pod="openshift-ingress-canary/ingress-canary-9snb9"
Jul 07 14:09:49.387581 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.387347    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b91a946-afd8-446e-af80-613ed4826f31-ca-trust-extracted\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.387713 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.387347    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/27acbc3b-ba1e-411b-a893-a78af377f3ad-config-volume\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.387713 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.387364    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-registry-certificates\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.387713 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.387492    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27acbc3b-ba1e-411b-a893-a78af377f3ad-tmp-dir\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.387713 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.387552    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 14:09:49.387713 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.387643    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f47d4797f-k8jfq: secret "image-registry-tls" not found
Jul 07 14:09:49.387713 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.387657    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-trusted-ca\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.387713 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.387607    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:49.388016 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.387695    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls podName:6b91a946-afd8-446e-af80-613ed4826f31 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:49.88768026 +0000 UTC m=+35.995991832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls") pod "image-registry-7f47d4797f-k8jfq" (UID: "6b91a946-afd8-446e-af80-613ed4826f31") : secret "image-registry-tls" not found
Jul 07 14:09:49.388016 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.387750    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls podName:27acbc3b-ba1e-411b-a893-a78af377f3ad nodeName:}" failed. No retries permitted until 2026-07-07 14:09:49.887738159 +0000 UTC m=+35.996049734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls") pod "dns-default-2kvq8" (UID: "27acbc3b-ba1e-411b-a893-a78af377f3ad") : secret "dns-default-metrics-tls" not found
Jul 07 14:09:49.391642 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.391612    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-installation-pull-secrets\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.391809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.391625    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41251dd9-eb69-46b2-a2c3-fbabac0dc673-cert\") pod \"ingress-canary-9snb9\" (UID: \"41251dd9-eb69-46b2-a2c3-fbabac0dc673\") " pod="openshift-ingress-canary/ingress-canary-9snb9"
Jul 07 14:09:49.391809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.391621    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-image-registry-private-configuration\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.402227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.402194    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8npt\" (UniqueName: \"kubernetes.io/projected/27acbc3b-ba1e-411b-a893-a78af377f3ad-kube-api-access-j8npt\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.402745 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.402714    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2799z\" (UniqueName: \"kubernetes.io/projected/41251dd9-eb69-46b2-a2c3-fbabac0dc673-kube-api-access-2799z\") pod \"ingress-canary-9snb9\" (UID: \"41251dd9-eb69-46b2-a2c3-fbabac0dc673\") " pod="openshift-ingress-canary/ingress-canary-9snb9"
Jul 07 14:09:49.402868 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.402810    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-bound-sa-token\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.403013 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.402997    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcbv5\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-kube-api-access-wcbv5\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.435128 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.435072    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:09:49.435316 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.435083    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:09:49.437868 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.437839    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:09:49.438049 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.438015    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:09:49.438281 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.438260    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8zlrf\""
Jul 07 14:09:49.438392 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.438371    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-66bkv\""
Jul 07 14:09:49.438658 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.438640    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:09:49.515288 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.515245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9snb9"
Jul 07 14:09:49.877730 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.877693    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9snb9"]
Jul 07 14:09:49.881972 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:09:49.881942    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41251dd9_eb69_46b2_a2c3_fbabac0dc673.slice/crio-cff0091b4ab7c842c7fdada66093dcda04166beb9c5618914cfaf20145c2be75 WatchSource:0}: Error finding container cff0091b4ab7c842c7fdada66093dcda04166beb9c5618914cfaf20145c2be75: Status 404 returned error can't find the container with id cff0091b4ab7c842c7fdada66093dcda04166beb9c5618914cfaf20145c2be75
Jul 07 14:09:49.891254 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.891222    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:49.891447 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:49.891289    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:49.891447 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.891363    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:49.891447 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.891398    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 14:09:49.891447 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.891414    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f47d4797f-k8jfq: secret "image-registry-tls" not found
Jul 07 14:09:49.891447 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.891417    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls podName:27acbc3b-ba1e-411b-a893-a78af377f3ad nodeName:}" failed. No retries permitted until 2026-07-07 14:09:50.89140064 +0000 UTC m=+36.999712212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls") pod "dns-default-2kvq8" (UID: "27acbc3b-ba1e-411b-a893-a78af377f3ad") : secret "dns-default-metrics-tls" not found
Jul 07 14:09:49.891624 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:49.891471    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls podName:6b91a946-afd8-446e-af80-613ed4826f31 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:50.891454821 +0000 UTC m=+36.999766417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls") pod "image-registry-7f47d4797f-k8jfq" (UID: "6b91a946-afd8-446e-af80-613ed4826f31") : secret "image-registry-tls" not found
Jul 07 14:09:50.435899 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:50.435866    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:09:50.438758 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:50.438722    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:09:50.604045 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:50.604002    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9snb9" event={"ID":"41251dd9-eb69-46b2-a2c3-fbabac0dc673","Type":"ContainerStarted","Data":"cff0091b4ab7c842c7fdada66093dcda04166beb9c5618914cfaf20145c2be75"}
Jul 07 14:09:50.607413 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:50.607380    2569 generic.go:356] "Generic (PLEG): container finished" podID="6cab4996-d1b9-4425-ba8a-c5889b50acfb" containerID="d42bdc6c0e7c23ac545ee4df9bc411aed17c9f47ec3052eb6345e593b52798ec" exitCode=0
Jul 07 14:09:50.607577 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:50.607445    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mkfq" event={"ID":"6cab4996-d1b9-4425-ba8a-c5889b50acfb","Type":"ContainerDied","Data":"d42bdc6c0e7c23ac545ee4df9bc411aed17c9f47ec3052eb6345e593b52798ec"}
Jul 07 14:09:50.899323 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:50.899290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:50.899529 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:50.899390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:50.899529 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:50.899475    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 14:09:50.899529 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:50.899496    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f47d4797f-k8jfq: secret "image-registry-tls" not found
Jul 07 14:09:50.899529 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:50.899496    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:50.899702 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:50.899562    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls podName:6b91a946-afd8-446e-af80-613ed4826f31 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:52.899542158 +0000 UTC m=+39.007853765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls") pod "image-registry-7f47d4797f-k8jfq" (UID: "6b91a946-afd8-446e-af80-613ed4826f31") : secret "image-registry-tls" not found
Jul 07 14:09:50.899702 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:50.899593    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls podName:27acbc3b-ba1e-411b-a893-a78af377f3ad nodeName:}" failed. No retries permitted until 2026-07-07 14:09:52.899572809 +0000 UTC m=+39.007884386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls") pod "dns-default-2kvq8" (UID: "27acbc3b-ba1e-411b-a893-a78af377f3ad") : secret "dns-default-metrics-tls" not found
Jul 07 14:09:51.484978 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:51.484934    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:51.607581 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:51.607329    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:51.616958 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:51.616919    2569 generic.go:356] "Generic (PLEG): container finished" podID="6cab4996-d1b9-4425-ba8a-c5889b50acfb" containerID="2828e885ba99ed259c08033b5923c516ad02d35d3cedaa40e423afca80ffa569" exitCode=0
Jul 07 14:09:51.617167 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:51.617007    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mkfq" event={"ID":"6cab4996-d1b9-4425-ba8a-c5889b50acfb","Type":"ContainerDied","Data":"2828e885ba99ed259c08033b5923c516ad02d35d3cedaa40e423afca80ffa569"}
Jul 07 14:09:52.343141 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.343076    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:52.444296 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.444263    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:52.621128 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.621078    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9snb9" event={"ID":"41251dd9-eb69-46b2-a2c3-fbabac0dc673","Type":"ContainerStarted","Data":"ca622c3c97aee3e5751cbf5c83ea401cf62cf22398fb89395bd7aa16fb998fc1"}
Jul 07 14:09:52.625071 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.625041    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mkfq" event={"ID":"6cab4996-d1b9-4425-ba8a-c5889b50acfb","Type":"ContainerStarted","Data":"b134c848a2c5885b2cd1efcd741632ef1833d2c40e55b54098757260597a6d38"}
Jul 07 14:09:52.642237 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.642091    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9snb9" podStartSLOduration=1.144104042 podStartE2EDuration="3.642066807s" totalImagesPullingTime="2.497962765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:52.641613701 +0000 UTC m=+38.749925296" watchObservedRunningTime="2026-07-07 14:09:52.642066807 +0000 UTC m=+38.750378403"
Jul 07 14:09:52.681334 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.681253    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4mkfq" podStartSLOduration=9.016348582 podStartE2EDuration="38.681228316s" totalImagesPullingTime="29.664879734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:52.679874681 +0000 UTC m=+38.788186276" watchObservedRunningTime="2026-07-07 14:09:52.681228316 +0000 UTC m=+38.789539913"
Jul 07 14:09:52.818172 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.818102    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:52.917846 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.916689    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:52.917846 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.916747    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:52.917846 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:52.916874    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 14:09:52.917846 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:52.916888    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f47d4797f-k8jfq: secret "image-registry-tls" not found
Jul 07 14:09:52.917846 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:52.916944    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls podName:6b91a946-afd8-446e-af80-613ed4826f31 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:56.916928813 +0000 UTC m=+43.025240398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls") pod "image-registry-7f47d4797f-k8jfq" (UID: "6b91a946-afd8-446e-af80-613ed4826f31") : secret "image-registry-tls" not found
Jul 07 14:09:52.917846 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:52.917392    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:52.917846 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:52.917433    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls podName:27acbc3b-ba1e-411b-a893-a78af377f3ad nodeName:}" failed. No retries permitted until 2026-07-07 14:09:56.917421292 +0000 UTC m=+43.025732863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls") pod "dns-default-2kvq8" (UID: "27acbc3b-ba1e-411b-a893-a78af377f3ad") : secret "dns-default-metrics-tls" not found
Jul 07 14:09:52.917846 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:52.917463    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:53.016467 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:53.016431    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:54.221377 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:54.221334    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:54.286330 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:54.286283    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:56.944220 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:56.944167    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:09:56.944611 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:56.944240    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:09:56.944611 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:56.944353    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:56.944611 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:56.944418    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls podName:27acbc3b-ba1e-411b-a893-a78af377f3ad nodeName:}" failed. No retries permitted until 2026-07-07 14:10:04.944402626 +0000 UTC m=+51.052714201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls") pod "dns-default-2kvq8" (UID: "27acbc3b-ba1e-411b-a893-a78af377f3ad") : secret "dns-default-metrics-tls" not found
Jul 07 14:09:56.944611 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:56.944356    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 14:09:56.944611 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:56.944462    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f47d4797f-k8jfq: secret "image-registry-tls" not found
Jul 07 14:09:56.944611 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:09:56.944502    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls podName:6b91a946-afd8-446e-af80-613ed4826f31 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:04.944491214 +0000 UTC m=+51.052802787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls") pod "image-registry-7f47d4797f-k8jfq" (UID: "6b91a946-afd8-446e-af80-613ed4826f31") : secret "image-registry-tls" not found
Jul 07 14:09:59.861124 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:59.861067    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:09:59.961279 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:09:59.961231    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:05.002440 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:05.002398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:10:05.002850 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:05.002489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:10:05.002850 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:05.002573    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:10:05.002850 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:05.002620    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls podName:27acbc3b-ba1e-411b-a893-a78af377f3ad nodeName:}" failed. No retries permitted until 2026-07-07 14:10:21.002606081 +0000 UTC m=+67.110917654 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls") pod "dns-default-2kvq8" (UID: "27acbc3b-ba1e-411b-a893-a78af377f3ad") : secret "dns-default-metrics-tls" not found
Jul 07 14:10:05.002850 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:05.002571    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 14:10:05.002850 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:05.002667    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f47d4797f-k8jfq: secret "image-registry-tls" not found
Jul 07 14:10:05.002850 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:05.002729    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls podName:6b91a946-afd8-446e-af80-613ed4826f31 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:21.002714624 +0000 UTC m=+67.111026196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls") pod "image-registry-7f47d4797f-k8jfq" (UID: "6b91a946-afd8-446e-af80-613ed4826f31") : secret "image-registry-tls" not found
Jul 07 14:10:09.291790 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:09.291747    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:09.395528 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:09.395493    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log" limit=1024
Jul 07 14:10:14.230924 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.230886    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64"]
Jul 07 14:10:14.262694 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.262654    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64"]
Jul 07 14:10:14.262855 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.262769    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64"
Jul 07 14:10:14.265599 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.265570    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.265599 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.265584    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.265802 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.265599    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kr2h7\""
Jul 07 14:10:14.335387 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.335356    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"]
Jul 07 14:10:14.349368 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.349329    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"]
Jul 07 14:10:14.349551 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.349466    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:14.351790 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.351762    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.351937 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.351838    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q54d5\""
Jul 07 14:10:14.352288 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.352270    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 07 14:10:14.352396 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.352376    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.366966 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.366927    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:14.367308 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.367285    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866bs\" (UniqueName: \"kubernetes.io/projected/8b8ff1b0-b894-4838-818f-829a14e7d677-kube-api-access-866bs\") pod \"volume-data-source-validator-5d456fdbcd-fff64\" (UID: \"8b8ff1b0-b894-4838-818f-829a14e7d677\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64"
Jul 07 14:10:14.367493 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.367468    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2prv\" (UniqueName: \"kubernetes.io/projected/d2231c68-f460-4fb9-b070-1aebb7bdbf53-kube-api-access-c2prv\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:14.371430 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.371398    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-rcshd"]
Jul 07 14:10:14.371549 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.371515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:14.374034 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.374003    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 07 14:10:14.374855 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.374833    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wsq9m\""
Jul 07 14:10:14.375030 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.375012    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.375095 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.375067    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.375165 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.375068    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 07 14:10:14.394195 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.394168    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"]
Jul 07 14:10:14.394195 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.394202    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"]
Jul 07 14:10:14.394495 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.394221    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-rcshd"]
Jul 07 14:10:14.394495 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.394293    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.396961 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.396941    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.397474 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.397455    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 07 14:10:14.397474 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.397470    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fxnzz\""
Jul 07 14:10:14.397612 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.397505    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.397612 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.397461    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 07 14:10:14.402703 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.402676    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 07 14:10:14.438368 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.438337    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"]
Jul 07 14:10:14.459700 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.459668    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"]
Jul 07 14:10:14.459855 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.459770    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.462310 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.462289    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.462466 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.462446    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 07 14:10:14.462542 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.462503    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.462542 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.462521    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zqvs9\""
Jul 07 14:10:14.462637 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.462454    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 07 14:10:14.467866 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.467844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866bs\" (UniqueName: \"kubernetes.io/projected/8b8ff1b0-b894-4838-818f-829a14e7d677-kube-api-access-866bs\") pod \"volume-data-source-validator-5d456fdbcd-fff64\" (UID: \"8b8ff1b0-b894-4838-818f-829a14e7d677\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64"
Jul 07 14:10:14.467999 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.467874    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806d472c-532b-49fa-a1d6-c2a047ae8c3b-trusted-ca\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.467999 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.467892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbtj\" (UniqueName: \"kubernetes.io/projected/5af4992e-4e6f-4445-938c-657fd6c085ab-kube-api-access-sgbtj\") pod \"service-ca-operator-5d46d9ffd4-9ksfh\" (UID: \"5af4992e-4e6f-4445-938c-657fd6c085ab\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.468127 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.467991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5af4992e-4e6f-4445-938c-657fd6c085ab-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-9ksfh\" (UID: \"5af4992e-4e6f-4445-938c-657fd6c085ab\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.468188 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.468123    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:14.468188 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.468164    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f52588df-a9cd-4b8a-b936-9bd1eff9002f-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:14.468293 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.468193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5af4992e-4e6f-4445-938c-657fd6c085ab-config\") pod \"service-ca-operator-5d46d9ffd4-9ksfh\" (UID: \"5af4992e-4e6f-4445-938c-657fd6c085ab\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.468293 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.468234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2prv\" (UniqueName: \"kubernetes.io/projected/d2231c68-f460-4fb9-b070-1aebb7bdbf53-kube-api-access-c2prv\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:14.468293 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.468259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/806d472c-532b-49fa-a1d6-c2a047ae8c3b-serving-cert\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.468391 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.468300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:14.468391 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.468357    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6z2f\" (UniqueName: \"kubernetes.io/projected/f52588df-a9cd-4b8a-b936-9bd1eff9002f-kube-api-access-h6z2f\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:14.468391 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:14.468385    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 14:10:14.468490 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.468394    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806d472c-532b-49fa-a1d6-c2a047ae8c3b-config\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.468490 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.468418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pvnp\" (UniqueName: \"kubernetes.io/projected/806d472c-532b-49fa-a1d6-c2a047ae8c3b-kube-api-access-7pvnp\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.468490 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:14.468442    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls podName:d2231c68-f460-4fb9-b070-1aebb7bdbf53 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:14.968421977 +0000 UTC m=+61.076733556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2czpl" (UID: "d2231c68-f460-4fb9-b070-1aebb7bdbf53") : secret "samples-operator-tls" not found
Jul 07 14:10:14.478898 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.478873    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.482717 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.482653    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2prv\" (UniqueName: \"kubernetes.io/projected/d2231c68-f460-4fb9-b070-1aebb7bdbf53-kube-api-access-c2prv\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:14.489923 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.489888    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.499342 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.499300    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866bs\" (UniqueName: \"kubernetes.io/projected/8b8ff1b0-b894-4838-818f-829a14e7d677-kube-api-access-866bs\") pod \"volume-data-source-validator-5d456fdbcd-fff64\" (UID: \"8b8ff1b0-b894-4838-818f-829a14e7d677\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64"
Jul 07 14:10:14.526900 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.526868    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q"]
Jul 07 14:10:14.543081 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.543045    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q"]
Jul 07 14:10:14.543241 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.543158    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q"
Jul 07 14:10:14.545400 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.545363    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jfjfb\""
Jul 07 14:10:14.548332 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.548301    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-596dffc677-vvdpt"]
Jul 07 14:10:14.569044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569012    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5af4992e-4e6f-4445-938c-657fd6c085ab-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-9ksfh\" (UID: \"5af4992e-4e6f-4445-938c-657fd6c085ab\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.569231 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569101    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:14.569231 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9rj5\" (UniqueName: \"kubernetes.io/projected/95d242cd-04c0-4e8a-9dee-48bcd0efc213-kube-api-access-p9rj5\") pod \"network-check-source-989f8b8dc-p5d4q\" (UID: \"95d242cd-04c0-4e8a-9dee-48bcd0efc213\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q"
Jul 07 14:10:14.569231 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569171    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f52588df-a9cd-4b8a-b936-9bd1eff9002f-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:14.569231 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5af4992e-4e6f-4445-938c-657fd6c085ab-config\") pod \"service-ca-operator-5d46d9ffd4-9ksfh\" (UID: \"5af4992e-4e6f-4445-938c-657fd6c085ab\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.569231 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569206    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/806d472c-532b-49fa-a1d6-c2a047ae8c3b-serving-cert\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.569231 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:14.569214    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:14.569519 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569249    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6z2f\" (UniqueName: \"kubernetes.io/projected/f52588df-a9cd-4b8a-b936-9bd1eff9002f-kube-api-access-h6z2f\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:14.569519 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:14.569293    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls podName:f52588df-a9cd-4b8a-b936-9bd1eff9002f nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.069268405 +0000 UTC m=+61.177580014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ldmlk" (UID: "f52588df-a9cd-4b8a-b936-9bd1eff9002f") : secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:14.569519 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569318    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806d472c-532b-49fa-a1d6-c2a047ae8c3b-config\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.569519 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569352    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pvnp\" (UniqueName: \"kubernetes.io/projected/806d472c-532b-49fa-a1d6-c2a047ae8c3b-kube-api-access-7pvnp\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.569519 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569387    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806d472c-532b-49fa-a1d6-c2a047ae8c3b-trusted-ca\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.569519 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.569414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbtj\" (UniqueName: \"kubernetes.io/projected/5af4992e-4e6f-4445-938c-657fd6c085ab-kube-api-access-sgbtj\") pod \"service-ca-operator-5d46d9ffd4-9ksfh\" (UID: \"5af4992e-4e6f-4445-938c-657fd6c085ab\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.570344 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.570008    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f52588df-a9cd-4b8a-b936-9bd1eff9002f-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:14.570344 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.570265    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806d472c-532b-49fa-a1d6-c2a047ae8c3b-config\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.570508 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.570388    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5af4992e-4e6f-4445-938c-657fd6c085ab-config\") pod \"service-ca-operator-5d46d9ffd4-9ksfh\" (UID: \"5af4992e-4e6f-4445-938c-657fd6c085ab\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.570508 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.570437    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"]
Jul 07 14:10:14.570627 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.570609    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.570660 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.570623    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806d472c-532b-49fa-a1d6-c2a047ae8c3b-trusted-ca\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.571819 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.571790    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5af4992e-4e6f-4445-938c-657fd6c085ab-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-9ksfh\" (UID: \"5af4992e-4e6f-4445-938c-657fd6c085ab\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.571917 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.571852    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/806d472c-532b-49fa-a1d6-c2a047ae8c3b-serving-cert\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.573207 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.573185    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 07 14:10:14.573207 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.573195    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lbqm2\""
Jul 07 14:10:14.573375 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.573229    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 07 14:10:14.573427 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.573377    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 07 14:10:14.573506 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.573480    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.573606 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.573554    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 07 14:10:14.573606 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.573594    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.573702 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.573688    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kr2h7\""
Jul 07 14:10:14.581712 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.581664    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64"
Jul 07 14:10:14.582695 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.582667    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbtj\" (UniqueName: \"kubernetes.io/projected/5af4992e-4e6f-4445-938c-657fd6c085ab-kube-api-access-sgbtj\") pod \"service-ca-operator-5d46d9ffd4-9ksfh\" (UID: \"5af4992e-4e6f-4445-938c-657fd6c085ab\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.582833 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.582809    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6z2f\" (UniqueName: \"kubernetes.io/projected/f52588df-a9cd-4b8a-b936-9bd1eff9002f-kube-api-access-h6z2f\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:14.583003 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.582910    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pvnp\" (UniqueName: \"kubernetes.io/projected/806d472c-532b-49fa-a1d6-c2a047ae8c3b-kube-api-access-7pvnp\") pod \"console-operator-68b757fb67-rcshd\" (UID: \"806d472c-532b-49fa-a1d6-c2a047ae8c3b\") " pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.604878 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.604843    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-k8t8j"]
Jul 07 14:10:14.605063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.604985    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.607532 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.607501    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 07 14:10:14.607733 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.607714    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 07 14:10:14.607952 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.607935    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-f7lf4\""
Jul 07 14:10:14.608022 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.608006    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.608203 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.608189    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.629031 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.629005    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzqcm"
Jul 07 14:10:14.629224 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.629062    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"]
Jul 07 14:10:14.629224 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.629085    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-k8t8j"]
Jul 07 14:10:14.629224 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.629121    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-596dffc677-vvdpt"]
Jul 07 14:10:14.629224 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.629148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.631537 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.631513    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.631537 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.631527    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.631805 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.631790    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 07 14:10:14.631856 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.631840    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 07 14:10:14.632032 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.632017    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c2ngh\""
Jul 07 14:10:14.637159 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.636886    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 07 14:10:14.670500 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670405    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-default-certificate\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.670500 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670453    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.670500 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85998ef5-484f-4de6-9dc5-7ded0e7317cd-snapshots\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.670678 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670509    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdqh\" (UniqueName: \"kubernetes.io/projected/85998ef5-484f-4de6-9dc5-7ded0e7317cd-kube-api-access-rxdqh\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.670678 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670588    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba2eb9cf-295b-4d9f-aee2-28ffa025e24c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-p4txj\" (UID: \"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.670678 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-stats-auth\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.670816 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba2eb9cf-295b-4d9f-aee2-28ffa025e24c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-p4txj\" (UID: \"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.670816 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85998ef5-484f-4de6-9dc5-7ded0e7317cd-tmp\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.670916 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.670916 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85998ef5-484f-4de6-9dc5-7ded0e7317cd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.671017 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lccsd\" (UniqueName: \"kubernetes.io/projected/75cb0ebd-abf1-489a-97e9-88324f9cf349-kube-api-access-lccsd\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.671017 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670960    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rj5\" (UniqueName: \"kubernetes.io/projected/95d242cd-04c0-4e8a-9dee-48bcd0efc213-kube-api-access-p9rj5\") pod \"network-check-source-989f8b8dc-p5d4q\" (UID: \"95d242cd-04c0-4e8a-9dee-48bcd0efc213\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q"
Jul 07 14:10:14.671017 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.670988    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsg77\" (UniqueName: \"kubernetes.io/projected/ba2eb9cf-295b-4d9f-aee2-28ffa025e24c-kube-api-access-vsg77\") pod \"kube-storage-version-migrator-operator-846bf668f7-p4txj\" (UID: \"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.671186 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.671020    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85998ef5-484f-4de6-9dc5-7ded0e7317cd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.671186 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.671057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85998ef5-484f-4de6-9dc5-7ded0e7317cd-serving-cert\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.688259 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.688007    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:10:14.699222 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.699192    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:10:14.707641 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.707569    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:14.710543 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.710503    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rj5\" (UniqueName: \"kubernetes.io/projected/95d242cd-04c0-4e8a-9dee-48bcd0efc213-kube-api-access-p9rj5\") pod \"network-check-source-989f8b8dc-p5d4q\" (UID: \"95d242cd-04c0-4e8a-9dee-48bcd0efc213\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q"
Jul 07 14:10:14.762595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.762556    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64"]
Jul 07 14:10:14.766309 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:14.766277    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8ff1b0_b894_4838_818f_829a14e7d677.slice/crio-cff7e74a2261cbdd7eb9800a2928a1a14f80eaa965c70787e5e28fa76f9ff9f6 WatchSource:0}: Error finding container cff7e74a2261cbdd7eb9800a2928a1a14f80eaa965c70787e5e28fa76f9ff9f6: Status 404 returned error can't find the container with id cff7e74a2261cbdd7eb9800a2928a1a14f80eaa965c70787e5e28fa76f9ff9f6
Jul 07 14:10:14.768684 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.768655    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"
Jul 07 14:10:14.771995 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.771968    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba2eb9cf-295b-4d9f-aee2-28ffa025e24c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-p4txj\" (UID: \"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.772147 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772015    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-stats-auth\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.772147 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772089    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba2eb9cf-295b-4d9f-aee2-28ffa025e24c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-p4txj\" (UID: \"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.772147 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772130    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85998ef5-484f-4de6-9dc5-7ded0e7317cd-tmp\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.772319 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.772319 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85998ef5-484f-4de6-9dc5-7ded0e7317cd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.772319 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772277    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lccsd\" (UniqueName: \"kubernetes.io/projected/75cb0ebd-abf1-489a-97e9-88324f9cf349-kube-api-access-lccsd\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.772702 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsg77\" (UniqueName: \"kubernetes.io/projected/ba2eb9cf-295b-4d9f-aee2-28ffa025e24c-kube-api-access-vsg77\") pod \"kube-storage-version-migrator-operator-846bf668f7-p4txj\" (UID: \"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.772702 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772349    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85998ef5-484f-4de6-9dc5-7ded0e7317cd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.772702 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85998ef5-484f-4de6-9dc5-7ded0e7317cd-serving-cert\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.772702 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-default-certificate\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.772702 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772441    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.772702 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772470    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85998ef5-484f-4de6-9dc5-7ded0e7317cd-snapshots\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.772702 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772497    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdqh\" (UniqueName: \"kubernetes.io/projected/85998ef5-484f-4de6-9dc5-7ded0e7317cd-kube-api-access-rxdqh\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.773053 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.772697    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba2eb9cf-295b-4d9f-aee2-28ffa025e24c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-p4txj\" (UID: \"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.773053 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:14.772787    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 14:10:14.773053 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:14.772840    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.272822456 +0000 UTC m=+61.381134042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : secret "router-metrics-certs-default" not found
Jul 07 14:10:14.773241 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.773182    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85998ef5-484f-4de6-9dc5-7ded0e7317cd-tmp\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.773241 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:14.773211    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.273190504 +0000 UTC m=+61.381502093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : configmap references non-existent config key: service-ca.crt
Jul 07 14:10:14.773711 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.773665    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85998ef5-484f-4de6-9dc5-7ded0e7317cd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.773903 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.773782    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85998ef5-484f-4de6-9dc5-7ded0e7317cd-snapshots\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.774581 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.774558    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85998ef5-484f-4de6-9dc5-7ded0e7317cd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.775585 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.775564    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba2eb9cf-295b-4d9f-aee2-28ffa025e24c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-p4txj\" (UID: \"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.776026 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.775975    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85998ef5-484f-4de6-9dc5-7ded0e7317cd-serving-cert\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.777035 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.777008    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-stats-auth\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.788540 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.788445    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-default-certificate\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.798258 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.797675    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsg77\" (UniqueName: \"kubernetes.io/projected/ba2eb9cf-295b-4d9f-aee2-28ffa025e24c-kube-api-access-vsg77\") pod \"kube-storage-version-migrator-operator-846bf668f7-p4txj\" (UID: \"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.804760 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.798517    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lccsd\" (UniqueName: \"kubernetes.io/projected/75cb0ebd-abf1-489a-97e9-88324f9cf349-kube-api-access-lccsd\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:14.804760 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.801757    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdqh\" (UniqueName: \"kubernetes.io/projected/85998ef5-484f-4de6-9dc5-7ded0e7317cd-kube-api-access-rxdqh\") pod \"insights-operator-5c847dcfc4-k8t8j\" (UID: \"85998ef5-484f-4de6-9dc5-7ded0e7317cd\") " pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.853578 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.853540    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q"
Jul 07 14:10:14.886904 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.886835    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-rcshd"]
Jul 07 14:10:14.890564 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:14.890460    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806d472c_532b_49fa_a1d6_c2a047ae8c3b.slice/crio-3b0ddc285733700f22135c92d45619d2bb66e41865904d239fbafb2d5c8e080e WatchSource:0}: Error finding container 3b0ddc285733700f22135c92d45619d2bb66e41865904d239fbafb2d5c8e080e: Status 404 returned error can't find the container with id 3b0ddc285733700f22135c92d45619d2bb66e41865904d239fbafb2d5c8e080e
Jul 07 14:10:14.914398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.914359    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"
Jul 07 14:10:14.940665 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.940585    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j"
Jul 07 14:10:14.949171 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.949080    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh"]
Jul 07 14:10:14.952781 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:14.952739    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af4992e_4e6f_4445_938c_657fd6c085ab.slice/crio-4183413f44d21b0a795df583ab2b437b74d1940a47f0a6f498c0844923b4fc60 WatchSource:0}: Error finding container 4183413f44d21b0a795df583ab2b437b74d1940a47f0a6f498c0844923b4fc60: Status 404 returned error can't find the container with id 4183413f44d21b0a795df583ab2b437b74d1940a47f0a6f498c0844923b4fc60
Jul 07 14:10:14.974720 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:14.974688    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:14.974882 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:14.974831    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 14:10:14.974923 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:14.974884    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls podName:d2231c68-f460-4fb9-b070-1aebb7bdbf53 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.97487073 +0000 UTC m=+62.083182303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2czpl" (UID: "d2231c68-f460-4fb9-b070-1aebb7bdbf53") : secret "samples-operator-tls" not found
Jul 07 14:10:15.025147 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.025087    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q"]
Jul 07 14:10:15.075975 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.075920    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:15.076161 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:15.076103    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:15.076223 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:15.076207    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls podName:f52588df-a9cd-4b8a-b936-9bd1eff9002f nodeName:}" failed. No retries permitted until 2026-07-07 14:10:16.076185267 +0000 UTC m=+62.184496871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ldmlk" (UID: "f52588df-a9cd-4b8a-b936-9bd1eff9002f") : secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:15.093584 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.093546    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj"]
Jul 07 14:10:15.096430 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:15.096400    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2eb9cf_295b_4d9f_aee2_28ffa025e24c.slice/crio-9e01e89a8d836e554a59c406443eff3a683851b4f045d4895ddc87618e769229 WatchSource:0}: Error finding container 9e01e89a8d836e554a59c406443eff3a683851b4f045d4895ddc87618e769229: Status 404 returned error can't find the container with id 9e01e89a8d836e554a59c406443eff3a683851b4f045d4895ddc87618e769229
Jul 07 14:10:15.112149 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.112099    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-k8t8j"]
Jul 07 14:10:15.115261 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:15.115231    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85998ef5_484f_4de6_9dc5_7ded0e7317cd.slice/crio-60e666d37b9397367f5e2b3c28a8a454198247abeceed11ca81083cac2646eae WatchSource:0}: Error finding container 60e666d37b9397367f5e2b3c28a8a454198247abeceed11ca81083cac2646eae: Status 404 returned error can't find the container with id 60e666d37b9397367f5e2b3c28a8a454198247abeceed11ca81083cac2646eae
Jul 07 14:10:15.278222 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.278105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:15.278596 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.278229    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:15.278596 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:15.278250    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 14:10:15.278596 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:15.278310    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:16.278295283 +0000 UTC m=+62.386606858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : secret "router-metrics-certs-default" not found
Jul 07 14:10:15.278596 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:15.278329    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:16.278317954 +0000 UTC m=+62.386629526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : configmap references non-existent config key: service-ca.crt
Jul 07 14:10:15.683732 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.683695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j" event={"ID":"85998ef5-484f-4de6-9dc5-7ded0e7317cd","Type":"ContainerStarted","Data":"60e666d37b9397367f5e2b3c28a8a454198247abeceed11ca81083cac2646eae"}
Jul 07 14:10:15.685136 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.685081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q" event={"ID":"95d242cd-04c0-4e8a-9dee-48bcd0efc213","Type":"ContainerStarted","Data":"d1d06b96759c4beb34c35e681c59a8c1c6789ccc103fbf102e81fd37fb68592b"}
Jul 07 14:10:15.686394 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.686361    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" event={"ID":"806d472c-532b-49fa-a1d6-c2a047ae8c3b","Type":"ContainerStarted","Data":"3b0ddc285733700f22135c92d45619d2bb66e41865904d239fbafb2d5c8e080e"}
Jul 07 14:10:15.687626 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.687601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj" event={"ID":"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c","Type":"ContainerStarted","Data":"9e01e89a8d836e554a59c406443eff3a683851b4f045d4895ddc87618e769229"}
Jul 07 14:10:15.688953 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.688930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64" event={"ID":"8b8ff1b0-b894-4838-818f-829a14e7d677","Type":"ContainerStarted","Data":"cff7e74a2261cbdd7eb9800a2928a1a14f80eaa965c70787e5e28fa76f9ff9f6"}
Jul 07 14:10:15.690104 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.690080    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh" event={"ID":"5af4992e-4e6f-4445-938c-657fd6c085ab","Type":"ContainerStarted","Data":"4183413f44d21b0a795df583ab2b437b74d1940a47f0a6f498c0844923b4fc60"}
Jul 07 14:10:15.984381 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:15.984082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:15.984381 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:15.984268    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 14:10:15.984381 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:15.984348    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls podName:d2231c68-f460-4fb9-b070-1aebb7bdbf53 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:17.984326397 +0000 UTC m=+64.092637969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2czpl" (UID: "d2231c68-f460-4fb9-b070-1aebb7bdbf53") : secret "samples-operator-tls" not found
Jul 07 14:10:16.085581 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:16.085535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:16.085778 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:16.085752    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:16.085841 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:16.085819    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls podName:f52588df-a9cd-4b8a-b936-9bd1eff9002f nodeName:}" failed. No retries permitted until 2026-07-07 14:10:18.085800156 +0000 UTC m=+64.194111730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ldmlk" (UID: "f52588df-a9cd-4b8a-b936-9bd1eff9002f") : secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:16.288046 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:16.287953    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:16.288545 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:16.288070    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:16.288545 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:16.288268    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:18.28824916 +0000 UTC m=+64.396560732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : configmap references non-existent config key: service-ca.crt
Jul 07 14:10:16.288668 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:16.288614    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 14:10:16.288726 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:16.288689    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:18.288671141 +0000 UTC m=+64.396982716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : secret "router-metrics-certs-default" not found
Jul 07 14:10:18.005639 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:18.005599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:18.006063 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:18.005784    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 14:10:18.006063 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:18.005875    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls podName:d2231c68-f460-4fb9-b070-1aebb7bdbf53 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:22.005849941 +0000 UTC m=+68.114161526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2czpl" (UID: "d2231c68-f460-4fb9-b070-1aebb7bdbf53") : secret "samples-operator-tls" not found
Jul 07 14:10:18.106717 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:18.106674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:18.106908 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:18.106813    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:18.106908 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:18.106890    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls podName:f52588df-a9cd-4b8a-b936-9bd1eff9002f nodeName:}" failed. No retries permitted until 2026-07-07 14:10:22.106869197 +0000 UTC m=+68.215180773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ldmlk" (UID: "f52588df-a9cd-4b8a-b936-9bd1eff9002f") : secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:18.308311 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:18.308216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:18.308466 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:18.308324    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:18.308466 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:18.308386    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 14:10:18.308466 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:18.308454    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:22.308441219 +0000 UTC m=+68.416752795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : configmap references non-existent config key: service-ca.crt
Jul 07 14:10:18.308466 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:18.308469    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:22.308463833 +0000 UTC m=+68.416775406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : secret "router-metrics-certs-default" not found
Jul 07 14:10:20.125946 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.125879    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:10:20.128736 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.128700    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:10:20.136826 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:20.136785    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 14:10:20.137005 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:20.136907    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs podName:4298379e-f6c7-44eb-b922-f393e007c4af nodeName:}" failed. No retries permitted until 2026-07-07 14:11:24.136883628 +0000 UTC m=+130.245195204 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs") pod "network-metrics-daemon-8jm6t" (UID: "4298379e-f6c7-44eb-b922-f393e007c4af") : secret "metrics-daemon-secret" not found
Jul 07 14:10:20.226819 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.226774    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:10:20.226980 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.226881    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:10:20.229589 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.229566    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdg86\" (UniqueName: \"kubernetes.io/projected/2ea869a2-8efc-4180-8f6a-5e4808dd6c28-kube-api-access-fdg86\") pod \"network-check-target-9r2vh\" (UID: \"2ea869a2-8efc-4180-8f6a-5e4808dd6c28\") " pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:10:20.229665 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.229638    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:10:20.239782 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.239746    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5f534d4-5cbf-4bbe-af34-41916ed91208-original-pull-secret\") pod \"global-pull-secret-syncer-zhwzv\" (UID: \"b5f534d4-5cbf-4bbe-af34-41916ed91208\") " pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:10:20.349559 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.349535    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8zlrf\""
Jul 07 14:10:20.357613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.357446    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:10:20.447604 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.447308    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhwzv"
Jul 07 14:10:20.569092 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.568953    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9r2vh"]
Jul 07 14:10:20.576043 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:20.575976    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea869a2_8efc_4180_8f6a_5e4808dd6c28.slice/crio-b3bc77778c364ba7f0f8b7a9d080850609b146788b48ae1a55ec67c66907c2dc WatchSource:0}: Error finding container b3bc77778c364ba7f0f8b7a9d080850609b146788b48ae1a55ec67c66907c2dc: Status 404 returned error can't find the container with id b3bc77778c364ba7f0f8b7a9d080850609b146788b48ae1a55ec67c66907c2dc
Jul 07 14:10:20.628864 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.628778    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zhwzv"]
Jul 07 14:10:20.636597 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:20.636560    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f534d4_5cbf_4bbe_af34_41916ed91208.slice/crio-365ddf56a025befa895e286f6b68ac89da794b97de25df1a0cae34789d0c21ea WatchSource:0}: Error finding container 365ddf56a025befa895e286f6b68ac89da794b97de25df1a0cae34789d0c21ea: Status 404 returned error can't find the container with id 365ddf56a025befa895e286f6b68ac89da794b97de25df1a0cae34789d0c21ea
Jul 07 14:10:20.710133 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.710075    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64" event={"ID":"8b8ff1b0-b894-4838-818f-829a14e7d677","Type":"ContainerStarted","Data":"ba9de7840e0dbcd23f8cad84774f70111c160152824b758c0890f96bd43c4190"}
Jul 07 14:10:20.711763 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.711727    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zhwzv" event={"ID":"b5f534d4-5cbf-4bbe-af34-41916ed91208","Type":"ContainerStarted","Data":"365ddf56a025befa895e286f6b68ac89da794b97de25df1a0cae34789d0c21ea"}
Jul 07 14:10:20.713690 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.713632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh" event={"ID":"5af4992e-4e6f-4445-938c-657fd6c085ab","Type":"ContainerStarted","Data":"44dfb7d644fde15fd176afd4ea16268f5c2b33985e0ce73dd1641576fcc8aca5"}
Jul 07 14:10:20.717161 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.717081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j" event={"ID":"85998ef5-484f-4de6-9dc5-7ded0e7317cd","Type":"ContainerStarted","Data":"978d2690dae7d5cfd85378ff075bb1142bddda0b56e6991c56dc847e56fee2e7"}
Jul 07 14:10:20.718835 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.718809    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q" event={"ID":"95d242cd-04c0-4e8a-9dee-48bcd0efc213","Type":"ContainerStarted","Data":"56ab42c46276958d290890d3abc4f5dc730aeafb919f0c82c8641cf4ece353ea"}
Jul 07 14:10:20.720575 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.720556    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/0.log"
Jul 07 14:10:20.720697 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.720600    2569 generic.go:356] "Generic (PLEG): container finished" podID="806d472c-532b-49fa-a1d6-c2a047ae8c3b" containerID="16672d1a00e862d04f9f823234a8c9b18154a550dac6d74d9235fc1ba38ea523" exitCode=255
Jul 07 14:10:20.720754 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.720683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" event={"ID":"806d472c-532b-49fa-a1d6-c2a047ae8c3b","Type":"ContainerDied","Data":"16672d1a00e862d04f9f823234a8c9b18154a550dac6d74d9235fc1ba38ea523"}
Jul 07 14:10:20.721275 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.721005    2569 scope.go:117] "RemoveContainer" containerID="16672d1a00e862d04f9f823234a8c9b18154a550dac6d74d9235fc1ba38ea523"
Jul 07 14:10:20.722723 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.722574    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9r2vh" event={"ID":"2ea869a2-8efc-4180-8f6a-5e4808dd6c28","Type":"ContainerStarted","Data":"bdef82db0c81fd03c4f1b904d6eb4854026ca17ed8f95ddf85adaa8e19b36ec7"}
Jul 07 14:10:20.722723 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.722622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9r2vh" event={"ID":"2ea869a2-8efc-4180-8f6a-5e4808dd6c28","Type":"ContainerStarted","Data":"b3bc77778c364ba7f0f8b7a9d080850609b146788b48ae1a55ec67c66907c2dc"}
Jul 07 14:10:20.722723 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.722670    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:10:20.724406 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.724374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj" event={"ID":"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c","Type":"ContainerStarted","Data":"c701a99b0e028f76365be71141d3e2a5e8f2cb30faf38f25d777eb8a1745170e"}
Jul 07 14:10:20.735238 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.735179    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fff64" podStartSLOduration=1.248663511 podStartE2EDuration="6.735155859s" totalImagesPullingTime="5.486492348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:20.734095897 +0000 UTC m=+66.842407492" watchObservedRunningTime="2026-07-07 14:10:20.735155859 +0000 UTC m=+66.843467454"
Jul 07 14:10:20.800927 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.800857    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9r2vh" podStartSLOduration=66.800835802 podStartE2EDuration="1m6.800835802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:20.769935286 +0000 UTC m=+66.878246881" watchObservedRunningTime="2026-07-07 14:10:20.800835802 +0000 UTC m=+66.909147445"
Jul 07 14:10:20.801390 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.801346    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh" podStartSLOduration=1.491662132 podStartE2EDuration="6.801329113s" totalImagesPullingTime="5.309666981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:20.799985635 +0000 UTC m=+66.908297229" watchObservedRunningTime="2026-07-07 14:10:20.801329113 +0000 UTC m=+66.909640707"
Jul 07 14:10:20.853185 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.853067    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p5d4q" podStartSLOduration=1.624649501 podStartE2EDuration="6.853044302s" totalImagesPullingTime="5.228394801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:20.852402028 +0000 UTC m=+66.960713823" watchObservedRunningTime="2026-07-07 14:10:20.853044302 +0000 UTC m=+66.961355896"
Jul 07 14:10:20.879537 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.879415    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj" podStartSLOduration=1.717585694 podStartE2EDuration="6.879391667s" totalImagesPullingTime="5.161805973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:20.878310766 +0000 UTC m=+66.986622361" watchObservedRunningTime="2026-07-07 14:10:20.879391667 +0000 UTC m=+66.987703262"
Jul 07 14:10:20.907753 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:20.907683    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j" podStartSLOduration=1.757764914 podStartE2EDuration="6.907658946s" totalImagesPullingTime="5.149894032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:20.906283639 +0000 UTC m=+67.014595234" watchObservedRunningTime="2026-07-07 14:10:20.907658946 +0000 UTC m=+67.015970547"
Jul 07 14:10:21.037086 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:21.037033    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:10:21.037308 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:21.037161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:10:21.037308 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:21.037219    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 14:10:21.037308 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:21.037245    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f47d4797f-k8jfq: secret "image-registry-tls" not found
Jul 07 14:10:21.037308 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:21.037277    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:10:21.037511 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:21.037319    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls podName:6b91a946-afd8-446e-af80-613ed4826f31 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:53.037297983 +0000 UTC m=+99.145609554 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls") pod "image-registry-7f47d4797f-k8jfq" (UID: "6b91a946-afd8-446e-af80-613ed4826f31") : secret "image-registry-tls" not found
Jul 07 14:10:21.037511 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:21.037338    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls podName:27acbc3b-ba1e-411b-a893-a78af377f3ad nodeName:}" failed. No retries permitted until 2026-07-07 14:10:53.037329578 +0000 UTC m=+99.145641159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls") pod "dns-default-2kvq8" (UID: "27acbc3b-ba1e-411b-a893-a78af377f3ad") : secret "dns-default-metrics-tls" not found
Jul 07 14:10:21.731330 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:21.731240    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/1.log"
Jul 07 14:10:21.732139 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:21.732039    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/0.log"
Jul 07 14:10:21.732139 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:21.732083    2569 generic.go:356] "Generic (PLEG): container finished" podID="806d472c-532b-49fa-a1d6-c2a047ae8c3b" containerID="5db90ea65ca4292bafef305ccb5e22b148b9362019d1b4aa0fce6206b5f72dad" exitCode=255
Jul 07 14:10:21.732962 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:21.732436    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" event={"ID":"806d472c-532b-49fa-a1d6-c2a047ae8c3b","Type":"ContainerDied","Data":"5db90ea65ca4292bafef305ccb5e22b148b9362019d1b4aa0fce6206b5f72dad"}
Jul 07 14:10:21.732962 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:21.732537    2569 scope.go:117] "RemoveContainer" containerID="16672d1a00e862d04f9f823234a8c9b18154a550dac6d74d9235fc1ba38ea523"
Jul 07 14:10:21.734009 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:21.733986    2569 scope.go:117] "RemoveContainer" containerID="5db90ea65ca4292bafef305ccb5e22b148b9362019d1b4aa0fce6206b5f72dad"
Jul 07 14:10:21.734206 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:21.734178    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-rcshd_openshift-console-operator(806d472c-532b-49fa-a1d6-c2a047ae8c3b)\"" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" podUID="806d472c-532b-49fa-a1d6-c2a047ae8c3b"
Jul 07 14:10:22.053492 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.053210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:22.053492 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.053363    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 14:10:22.053492 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.053425    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls podName:d2231c68-f460-4fb9-b070-1aebb7bdbf53 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:30.053410002 +0000 UTC m=+76.161721574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2czpl" (UID: "d2231c68-f460-4fb9-b070-1aebb7bdbf53") : secret "samples-operator-tls" not found
Jul 07 14:10:22.054790 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.054761    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-45gjt"]
Jul 07 14:10:22.076711 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.076629    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.083683 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.083412    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xhpqs\""
Jul 07 14:10:22.087648 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.087425    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 14:10:22.087784 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.087721    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 14:10:22.093150 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.093098    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-45gjt"]
Jul 07 14:10:22.154194 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.154145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.154391 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.154225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6aadd26c-2770-473f-8987-aed718ce85d1-crio-socket\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.154391 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.154259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.154391 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.154306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6aadd26c-2770-473f-8987-aed718ce85d1-data-volume\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.154391 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.154331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfzq\" (UniqueName: \"kubernetes.io/projected/6aadd26c-2770-473f-8987-aed718ce85d1-kube-api-access-fqfzq\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.154589 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.154415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:22.154589 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.154559    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:22.154687 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.154621    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls podName:f52588df-a9cd-4b8a-b936-9bd1eff9002f nodeName:}" failed. No retries permitted until 2026-07-07 14:10:30.154601242 +0000 UTC m=+76.262912828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ldmlk" (UID: "f52588df-a9cd-4b8a-b936-9bd1eff9002f") : secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:22.255942 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.255504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6aadd26c-2770-473f-8987-aed718ce85d1-crio-socket\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.255942 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.255562    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.255942 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.255648    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:10:22.255942 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.255688    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6aadd26c-2770-473f-8987-aed718ce85d1-crio-socket\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.255942 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.255709    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:22.755691091 +0000 UTC m=+68.864002681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : configmap "kube-rbac-proxy" not found
Jul 07 14:10:22.255942 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.255761    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6aadd26c-2770-473f-8987-aed718ce85d1-data-volume\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.255942 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.255790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfzq\" (UniqueName: \"kubernetes.io/projected/6aadd26c-2770-473f-8987-aed718ce85d1-kube-api-access-fqfzq\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.256470 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.256213    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6aadd26c-2770-473f-8987-aed718ce85d1-data-volume\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.256470 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.256367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.256574 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.256479    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:22.256574 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.256527    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:22.756512324 +0000 UTC m=+68.864823904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:22.271081 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.271009    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfzq\" (UniqueName: \"kubernetes.io/projected/6aadd26c-2770-473f-8987-aed718ce85d1-kube-api-access-fqfzq\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.357390 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.357304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:22.357618 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.357402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:22.357618 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.357465    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 14:10:22.357618 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.357541    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:30.357521229 +0000 UTC m=+76.465832821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : secret "router-metrics-certs-default" not found
Jul 07 14:10:22.357618 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.357561    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:30.357551246 +0000 UTC m=+76.465862832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : configmap references non-existent config key: service-ca.crt
Jul 07 14:10:22.738136 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.738087    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/1.log"
Jul 07 14:10:22.738729 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.738709    2569 scope.go:117] "RemoveContainer" containerID="5db90ea65ca4292bafef305ccb5e22b148b9362019d1b4aa0fce6206b5f72dad"
Jul 07 14:10:22.739004 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.738913    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-rcshd_openshift-console-operator(806d472c-532b-49fa-a1d6-c2a047ae8c3b)\"" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" podUID="806d472c-532b-49fa-a1d6-c2a047ae8c3b"
Jul 07 14:10:22.761093 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.761054    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.761287 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:22.761160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:22.761287 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.761224    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:10:22.761397 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.761290    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:23.761264131 +0000 UTC m=+69.869575704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : configmap "kube-rbac-proxy" not found
Jul 07 14:10:22.761397 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.761227    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:22.761397 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:22.761383    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:23.761358522 +0000 UTC m=+69.869670116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:23.634944 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:23.634909    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cfxrl_4963cb5e-680a-4bbf-8e68-258d0f5b7331/dns-node-resolver/0.log"
Jul 07 14:10:23.771563 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:23.771520    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:23.772036 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:23.771608    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:23.772036 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:23.771711    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:23.772036 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:23.771762    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:10:23.772036 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:23.771792    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:25.771769328 +0000 UTC m=+71.880080905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:23.772036 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:23.771818    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:25.771800579 +0000 UTC m=+71.880112173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : configmap "kube-rbac-proxy" not found
Jul 07 14:10:24.708131 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:24.708063    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:24.708315 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:24.708163    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:24.708627 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:24.708601    2569 scope.go:117] "RemoveContainer" containerID="5db90ea65ca4292bafef305ccb5e22b148b9362019d1b4aa0fce6206b5f72dad"
Jul 07 14:10:24.708824 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:24.708795    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-rcshd_openshift-console-operator(806d472c-532b-49fa-a1d6-c2a047ae8c3b)\"" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" podUID="806d472c-532b-49fa-a1d6-c2a047ae8c3b"
Jul 07 14:10:24.835393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:24.835364    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5pz8_70208e8b-d198-4a27-abb8-248b9a6b9234/node-ca/0.log"
Jul 07 14:10:25.252395 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:25.252363    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9snb9_41251dd9-eb69-46b2-a2c3-fbabac0dc673/serve-healthcheck-canary/0.log"
Jul 07 14:10:25.752179 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:25.752142    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zhwzv" event={"ID":"b5f534d4-5cbf-4bbe-af34-41916ed91208","Type":"ContainerStarted","Data":"d82aaabb97c6905fb1132305c4838cf7cfdda2c0ce38a7f2e0bd874bed77a7d9"}
Jul 07 14:10:25.783600 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:25.783542    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zhwzv" podStartSLOduration=67.63881297 podStartE2EDuration="1m11.783525383s" totalImagesPullingTime="4.144712413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:25.782272464 +0000 UTC m=+71.890584069" watchObservedRunningTime="2026-07-07 14:10:25.783525383 +0000 UTC m=+71.891836999"
Jul 07 14:10:25.790709 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:25.790678    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:25.790830 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:25.790807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:25.790870 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:25.790820    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:10:25.790909 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:25.790885    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:29.790868499 +0000 UTC m=+75.899180074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : configmap "kube-rbac-proxy" not found
Jul 07 14:10:25.790955 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:25.790930    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:25.791001 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:25.790983    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:29.7909715 +0000 UTC m=+75.899283072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:29.829135 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:29.829074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:29.829632 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:29.829211    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:29.829632 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:29.829251    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:29.829632 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:29.829327    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:10:29.829632 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:29.829330    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:37.82931241 +0000 UTC m=+83.937623999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:29.829632 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:29.829396    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:37.829378911 +0000 UTC m=+83.937690488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : configmap "kube-rbac-proxy" not found
Jul 07 14:10:30.131529 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:30.131436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:30.131693 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:30.131557    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 14:10:30.131693 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:30.131612    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls podName:d2231c68-f460-4fb9-b070-1aebb7bdbf53 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:46.131597006 +0000 UTC m=+92.239908578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2czpl" (UID: "d2231c68-f460-4fb9-b070-1aebb7bdbf53") : secret "samples-operator-tls" not found
Jul 07 14:10:30.232617 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:30.232575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:30.232783 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:30.232745    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:30.232826 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:30.232812    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls podName:f52588df-a9cd-4b8a-b936-9bd1eff9002f nodeName:}" failed. No retries permitted until 2026-07-07 14:10:46.232794176 +0000 UTC m=+92.341105748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ldmlk" (UID: "f52588df-a9cd-4b8a-b936-9bd1eff9002f") : secret "cluster-monitoring-operator-tls" not found
Jul 07 14:10:30.434322 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:30.434278    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:30.434540 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:30.434356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:30.434540 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:30.434462    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 14:10:30.434540 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:30.434529    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:46.434511264 +0000 UTC m=+92.542822842 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : configmap references non-existent config key: service-ca.crt
Jul 07 14:10:30.434689 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:30.434551    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs podName:75cb0ebd-abf1-489a-97e9-88324f9cf349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:46.434543983 +0000 UTC m=+92.542855555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs") pod "router-default-596dffc677-vvdpt" (UID: "75cb0ebd-abf1-489a-97e9-88324f9cf349") : secret "router-metrics-certs-default" not found
Jul 07 14:10:37.435221 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.435188    2569 scope.go:117] "RemoveContainer" containerID="5db90ea65ca4292bafef305ccb5e22b148b9362019d1b4aa0fce6206b5f72dad"
Jul 07 14:10:37.795376 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.795284    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:10:37.795721 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.795704    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/1.log"
Jul 07 14:10:37.795777 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.795740    2569 generic.go:356] "Generic (PLEG): container finished" podID="806d472c-532b-49fa-a1d6-c2a047ae8c3b" containerID="e33107cd6bc948b6553dd9ef242a4cdb8d38bf0af36311730f17bc354c7f88b3" exitCode=255
Jul 07 14:10:37.795813 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.795786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" event={"ID":"806d472c-532b-49fa-a1d6-c2a047ae8c3b","Type":"ContainerDied","Data":"e33107cd6bc948b6553dd9ef242a4cdb8d38bf0af36311730f17bc354c7f88b3"}
Jul 07 14:10:37.795875 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.795862    2569 scope.go:117] "RemoveContainer" containerID="5db90ea65ca4292bafef305ccb5e22b148b9362019d1b4aa0fce6206b5f72dad"
Jul 07 14:10:37.796306 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.796290    2569 scope.go:117] "RemoveContainer" containerID="e33107cd6bc948b6553dd9ef242a4cdb8d38bf0af36311730f17bc354c7f88b3"
Jul 07 14:10:37.796477 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:37.796458    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-rcshd_openshift-console-operator(806d472c-532b-49fa-a1d6-c2a047ae8c3b)\"" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" podUID="806d472c-532b-49fa-a1d6-c2a047ae8c3b"
Jul 07 14:10:37.908086 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.908047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:37.908323 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:37.908237    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:37.908323 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.908299    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:37.908442 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:37.908343    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls podName:6aadd26c-2770-473f-8987-aed718ce85d1 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:53.90832363 +0000 UTC m=+100.016635214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45gjt" (UID: "6aadd26c-2770-473f-8987-aed718ce85d1") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:10:37.908871 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:37.908850    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aadd26c-2770-473f-8987-aed718ce85d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:38.800235 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:38.800203    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:10:44.708760 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:44.708716    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:44.709271 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:44.708808    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:10:44.709271 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:44.709200    2569 scope.go:117] "RemoveContainer" containerID="e33107cd6bc948b6553dd9ef242a4cdb8d38bf0af36311730f17bc354c7f88b3"
Jul 07 14:10:44.709403 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:44.709378    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-rcshd_openshift-console-operator(806d472c-532b-49fa-a1d6-c2a047ae8c3b)\"" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" podUID="806d472c-532b-49fa-a1d6-c2a047ae8c3b"
Jul 07 14:10:46.184919 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.184875    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:46.187488 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.187461    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2231c68-f460-4fb9-b070-1aebb7bdbf53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2czpl\" (UID: \"d2231c68-f460-4fb9-b070-1aebb7bdbf53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:46.286299 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.286259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:46.288743 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.288714    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52588df-a9cd-4b8a-b936-9bd1eff9002f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ldmlk\" (UID: \"f52588df-a9cd-4b8a-b936-9bd1eff9002f\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:46.460169 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.460055    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"
Jul 07 14:10:46.480353 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.480318    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"
Jul 07 14:10:46.487917 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.487882    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:46.488064 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.487948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:46.488564 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.488540    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75cb0ebd-abf1-489a-97e9-88324f9cf349-service-ca-bundle\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:46.490466 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.490434    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75cb0ebd-abf1-489a-97e9-88324f9cf349-metrics-certs\") pod \"router-default-596dffc677-vvdpt\" (UID: \"75cb0ebd-abf1-489a-97e9-88324f9cf349\") " pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:46.612658 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.612629    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl"]
Jul 07 14:10:46.630347 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.630316    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk"]
Jul 07 14:10:46.633791 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:46.633750    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52588df_a9cd_4b8a_b936_9bd1eff9002f.slice/crio-92dd62e0a0d60af414175c3d9b465c09a9cf8e05d02b107fcb2bd9c45227179b WatchSource:0}: Error finding container 92dd62e0a0d60af414175c3d9b465c09a9cf8e05d02b107fcb2bd9c45227179b: Status 404 returned error can't find the container with id 92dd62e0a0d60af414175c3d9b465c09a9cf8e05d02b107fcb2bd9c45227179b
Jul 07 14:10:46.691272 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.691233    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:46.827190 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.827151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk" event={"ID":"f52588df-a9cd-4b8a-b936-9bd1eff9002f","Type":"ContainerStarted","Data":"92dd62e0a0d60af414175c3d9b465c09a9cf8e05d02b107fcb2bd9c45227179b"}
Jul 07 14:10:46.828238 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.828211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl" event={"ID":"d2231c68-f460-4fb9-b070-1aebb7bdbf53","Type":"ContainerStarted","Data":"f3baebe3e586dc88d0f6c0d7985bc7931df4164ce08fbbcab689263ca0a3839a"}
Jul 07 14:10:46.833744 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:46.833717    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-596dffc677-vvdpt"]
Jul 07 14:10:46.836764 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:46.836737    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cb0ebd_abf1_489a_97e9_88324f9cf349.slice/crio-3c84e114065708c30542afc5307732c68e2b0ab23d0ff9c8a4746a7eff249ff1 WatchSource:0}: Error finding container 3c84e114065708c30542afc5307732c68e2b0ab23d0ff9c8a4746a7eff249ff1: Status 404 returned error can't find the container with id 3c84e114065708c30542afc5307732c68e2b0ab23d0ff9c8a4746a7eff249ff1
Jul 07 14:10:47.836478 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:47.836438    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-596dffc677-vvdpt" event={"ID":"75cb0ebd-abf1-489a-97e9-88324f9cf349","Type":"ContainerStarted","Data":"83d7ed90e5e1395ce447ba72017d20d2d90e04a40d2c3ae42f09ec3312490585"}
Jul 07 14:10:47.836971 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:47.836493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-596dffc677-vvdpt" event={"ID":"75cb0ebd-abf1-489a-97e9-88324f9cf349","Type":"ContainerStarted","Data":"3c84e114065708c30542afc5307732c68e2b0ab23d0ff9c8a4746a7eff249ff1"}
Jul 07 14:10:48.692713 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:48.692658    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:48.695494 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:48.695466    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:48.718361 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:48.718294    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-596dffc677-vvdpt" podStartSLOduration=34.718270034 podStartE2EDuration="34.718270034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:47.863720744 +0000 UTC m=+93.972032339" watchObservedRunningTime="2026-07-07 14:10:48.718270034 +0000 UTC m=+94.826581653"
Jul 07 14:10:48.841656 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:48.841612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl" event={"ID":"d2231c68-f460-4fb9-b070-1aebb7bdbf53","Type":"ContainerStarted","Data":"d2870616b3add16d0894d96dfecc373fd38401ad72b784dc3e969079c72e7cde"}
Jul 07 14:10:48.842148 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:48.841888    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:48.843447 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:48.843420    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-596dffc677-vvdpt"
Jul 07 14:10:49.847309 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.847268    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl" event={"ID":"d2231c68-f460-4fb9-b070-1aebb7bdbf53","Type":"ContainerStarted","Data":"0ad881c35e9768275911076cd1f6d960dba46ce113313e118ff147f91321b1e4"}
Jul 07 14:10:49.848798 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.848774    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk" event={"ID":"f52588df-a9cd-4b8a-b936-9bd1eff9002f","Type":"ContainerStarted","Data":"8282a37f1cd053e04af8df4ebf5b0d55cbd56136f2aafc76c98a00077c9e6eb3"}
Jul 07 14:10:49.871508 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.871451    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2czpl" podStartSLOduration=33.935451731 podStartE2EDuration="35.871432348s" totalImagesPullingTime="1.935980617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:49.870998855 +0000 UTC m=+95.979310450" watchObservedRunningTime="2026-07-07 14:10:49.871432348 +0000 UTC m=+95.979743997"
Jul 07 14:10:49.897906 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.897855    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ldmlk" podStartSLOduration=33.350167144 podStartE2EDuration="35.897835136s" totalImagesPullingTime="2.547667992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:49.897480486 +0000 UTC m=+96.005792082" watchObservedRunningTime="2026-07-07 14:10:49.897835136 +0000 UTC m=+96.006146730"
Jul 07 14:10:49.953047 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.953013    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"]
Jul 07 14:10:49.959685 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.959656    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"]
Jul 07 14:10:49.959845 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.959790    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:49.962398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.962373    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"
Jul 07 14:10:49.965575 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.965549    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 14:10:49.966589 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.966566    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 14:10:49.966689 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.966588    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 14:10:49.968989 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.968969    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 14:10:49.978609 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.978585    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6xffk\""
Jul 07 14:10:49.979224 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.979205    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 07 14:10:49.980079 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.980058    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"]
Jul 07 14:10:49.989732 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:49.989692    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"]
Jul 07 14:10:50.017061 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.017018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q647\" (UniqueName: \"kubernetes.io/projected/954d984a-0229-4a52-9f7a-6daf62db747d-kube-api-access-9q647\") pod \"klusterlet-addon-workmgr-6587794879-vmd5w\" (UID: \"954d984a-0229-4a52-9f7a-6daf62db747d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.017061 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.017059    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/954d984a-0229-4a52-9f7a-6daf62db747d-tmp\") pod \"klusterlet-addon-workmgr-6587794879-vmd5w\" (UID: \"954d984a-0229-4a52-9f7a-6daf62db747d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.017337 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.017128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmxrl\" (UniqueName: \"kubernetes.io/projected/ce4050d8-e32e-4836-94e7-bda9922917dd-kube-api-access-zmxrl\") pod \"managed-serviceaccount-addon-agent-6d5c99f576-2g548\" (UID: \"ce4050d8-e32e-4836-94e7-bda9922917dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"
Jul 07 14:10:50.017337 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.017162    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/954d984a-0229-4a52-9f7a-6daf62db747d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6587794879-vmd5w\" (UID: \"954d984a-0229-4a52-9f7a-6daf62db747d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.017337 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.017258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce4050d8-e32e-4836-94e7-bda9922917dd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d5c99f576-2g548\" (UID: \"ce4050d8-e32e-4836-94e7-bda9922917dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"
Jul 07 14:10:50.118664 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.118571    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmxrl\" (UniqueName: \"kubernetes.io/projected/ce4050d8-e32e-4836-94e7-bda9922917dd-kube-api-access-zmxrl\") pod \"managed-serviceaccount-addon-agent-6d5c99f576-2g548\" (UID: \"ce4050d8-e32e-4836-94e7-bda9922917dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"
Jul 07 14:10:50.118664 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.118643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/954d984a-0229-4a52-9f7a-6daf62db747d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6587794879-vmd5w\" (UID: \"954d984a-0229-4a52-9f7a-6daf62db747d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.118877 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.118728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce4050d8-e32e-4836-94e7-bda9922917dd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d5c99f576-2g548\" (UID: \"ce4050d8-e32e-4836-94e7-bda9922917dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"
Jul 07 14:10:50.118877 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.118789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q647\" (UniqueName: \"kubernetes.io/projected/954d984a-0229-4a52-9f7a-6daf62db747d-kube-api-access-9q647\") pod \"klusterlet-addon-workmgr-6587794879-vmd5w\" (UID: \"954d984a-0229-4a52-9f7a-6daf62db747d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.118877 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.118816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/954d984a-0229-4a52-9f7a-6daf62db747d-tmp\") pod \"klusterlet-addon-workmgr-6587794879-vmd5w\" (UID: \"954d984a-0229-4a52-9f7a-6daf62db747d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.119261 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.119237    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/954d984a-0229-4a52-9f7a-6daf62db747d-tmp\") pod \"klusterlet-addon-workmgr-6587794879-vmd5w\" (UID: \"954d984a-0229-4a52-9f7a-6daf62db747d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.121268 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.121247    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/954d984a-0229-4a52-9f7a-6daf62db747d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6587794879-vmd5w\" (UID: \"954d984a-0229-4a52-9f7a-6daf62db747d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.121369 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.121299    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce4050d8-e32e-4836-94e7-bda9922917dd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d5c99f576-2g548\" (UID: \"ce4050d8-e32e-4836-94e7-bda9922917dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"
Jul 07 14:10:50.133988 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.133949    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmxrl\" (UniqueName: \"kubernetes.io/projected/ce4050d8-e32e-4836-94e7-bda9922917dd-kube-api-access-zmxrl\") pod \"managed-serviceaccount-addon-agent-6d5c99f576-2g548\" (UID: \"ce4050d8-e32e-4836-94e7-bda9922917dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"
Jul 07 14:10:50.152917 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.143728    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q647\" (UniqueName: \"kubernetes.io/projected/954d984a-0229-4a52-9f7a-6daf62db747d-kube-api-access-9q647\") pod \"klusterlet-addon-workmgr-6587794879-vmd5w\" (UID: \"954d984a-0229-4a52-9f7a-6daf62db747d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.270041 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.270002    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:50.282914 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.282879    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"
Jul 07 14:10:50.498416 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.498389    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"]
Jul 07 14:10:50.499842 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:50.499815    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce4050d8_e32e_4836_94e7_bda9922917dd.slice/crio-57954faf4e5151ff6636cc5fd7d3cd2f84cffb1685debc70812a873c73f33d77 WatchSource:0}: Error finding container 57954faf4e5151ff6636cc5fd7d3cd2f84cffb1685debc70812a873c73f33d77: Status 404 returned error can't find the container with id 57954faf4e5151ff6636cc5fd7d3cd2f84cffb1685debc70812a873c73f33d77
Jul 07 14:10:50.500806 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:50.500782    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954d984a_0229_4a52_9f7a_6daf62db747d.slice/crio-9ce618eee1da5b5234170b92ee9d6dff1ce8e1da54b200d11371b505791f0ab5 WatchSource:0}: Error finding container 9ce618eee1da5b5234170b92ee9d6dff1ce8e1da54b200d11371b505791f0ab5: Status 404 returned error can't find the container with id 9ce618eee1da5b5234170b92ee9d6dff1ce8e1da54b200d11371b505791f0ab5
Jul 07 14:10:50.501173 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.501149    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548"]
Jul 07 14:10:50.854944 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.854906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w" event={"ID":"954d984a-0229-4a52-9f7a-6daf62db747d","Type":"ContainerStarted","Data":"9ce618eee1da5b5234170b92ee9d6dff1ce8e1da54b200d11371b505791f0ab5"}
Jul 07 14:10:50.856158 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:50.856133    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548" event={"ID":"ce4050d8-e32e-4836-94e7-bda9922917dd","Type":"ContainerStarted","Data":"57954faf4e5151ff6636cc5fd7d3cd2f84cffb1685debc70812a873c73f33d77"}
Jul 07 14:10:51.737304 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:51.737254    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9r2vh"
Jul 07 14:10:53.048571 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.048526    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:10:53.049017 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.048732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:10:53.051958 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.051927    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"image-registry-7f47d4797f-k8jfq\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") " pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:10:53.053049 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.053017    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27acbc3b-ba1e-411b-a893-a78af377f3ad-metrics-tls\") pod \"dns-default-2kvq8\" (UID: \"27acbc3b-ba1e-411b-a893-a78af377f3ad\") " pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:10:53.090735 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.090697    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bstgj\""
Jul 07 14:10:53.099314 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.099188    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:10:53.099697 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.099663    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-njxhm\""
Jul 07 14:10:53.107588 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.107549    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:10:53.956466 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.956421    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:53.959044 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:53.959012    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aadd26c-2770-473f-8987-aed718ce85d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45gjt\" (UID: \"6aadd26c-2770-473f-8987-aed718ce85d1\") " pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:54.193840 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:54.193799    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-45gjt"
Jul 07 14:10:55.256148 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.255665    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f47d4797f-k8jfq"]
Jul 07 14:10:55.260555 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:55.260510    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b91a946_afd8_446e_af80_613ed4826f31.slice/crio-90f9d0d1ad2cfacb037611aa6832ee14cd79c48c5857af7fa946f872b7081e5a WatchSource:0}: Error finding container 90f9d0d1ad2cfacb037611aa6832ee14cd79c48c5857af7fa946f872b7081e5a: Status 404 returned error can't find the container with id 90f9d0d1ad2cfacb037611aa6832ee14cd79c48c5857af7fa946f872b7081e5a
Jul 07 14:10:55.435412 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.435377    2569 scope.go:117] "RemoveContainer" containerID="e33107cd6bc948b6553dd9ef242a4cdb8d38bf0af36311730f17bc354c7f88b3"
Jul 07 14:10:55.435601 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:10:55.435565    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-rcshd_openshift-console-operator(806d472c-532b-49fa-a1d6-c2a047ae8c3b)\"" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" podUID="806d472c-532b-49fa-a1d6-c2a047ae8c3b"
Jul 07 14:10:55.492260 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.487649    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2kvq8"]
Jul 07 14:10:55.492260 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.489868    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-45gjt"]
Jul 07 14:10:55.494342 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:55.494271    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aadd26c_2770_473f_8987_aed718ce85d1.slice/crio-1c3fa8c762e1e9bbbdda31ef13943c1f9de32de9ed26cf3b86471f3c3528d5e8 WatchSource:0}: Error finding container 1c3fa8c762e1e9bbbdda31ef13943c1f9de32de9ed26cf3b86471f3c3528d5e8: Status 404 returned error can't find the container with id 1c3fa8c762e1e9bbbdda31ef13943c1f9de32de9ed26cf3b86471f3c3528d5e8
Jul 07 14:10:55.495371 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:10:55.495347    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27acbc3b_ba1e_411b_a893_a78af377f3ad.slice/crio-ea6f903821c4511da3b157b6bdb558997c6ef95a2109bbcbeb6292c752af2ad2 WatchSource:0}: Error finding container ea6f903821c4511da3b157b6bdb558997c6ef95a2109bbcbeb6292c752af2ad2: Status 404 returned error can't find the container with id ea6f903821c4511da3b157b6bdb558997c6ef95a2109bbcbeb6292c752af2ad2
Jul 07 14:10:55.882206 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.882155    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2kvq8" event={"ID":"27acbc3b-ba1e-411b-a893-a78af377f3ad","Type":"ContainerStarted","Data":"ea6f903821c4511da3b157b6bdb558997c6ef95a2109bbcbeb6292c752af2ad2"}
Jul 07 14:10:55.883753 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.883715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548" event={"ID":"ce4050d8-e32e-4836-94e7-bda9922917dd","Type":"ContainerStarted","Data":"d8ec165adfb927b2f595f852225046cc8f185c574658aacf06c02fe718bd95c1"}
Jul 07 14:10:55.885210 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.885181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w" event={"ID":"954d984a-0229-4a52-9f7a-6daf62db747d","Type":"ContainerStarted","Data":"cf1b1dec2f8a4e6dc044a65644bcff17ddb88e42965b971642fa8545c0b0a73a"}
Jul 07 14:10:55.885342 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.885324    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:55.886605 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.886571    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-45gjt" event={"ID":"6aadd26c-2770-473f-8987-aed718ce85d1","Type":"ContainerStarted","Data":"1c3fa8c762e1e9bbbdda31ef13943c1f9de32de9ed26cf3b86471f3c3528d5e8"}
Jul 07 14:10:55.887371 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.887348    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w"
Jul 07 14:10:55.888348 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.888327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq" event={"ID":"6b91a946-afd8-446e-af80-613ed4826f31","Type":"ContainerStarted","Data":"cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18"}
Jul 07 14:10:55.888450 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.888359    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq" event={"ID":"6b91a946-afd8-446e-af80-613ed4826f31","Type":"ContainerStarted","Data":"90f9d0d1ad2cfacb037611aa6832ee14cd79c48c5857af7fa946f872b7081e5a"}
Jul 07 14:10:55.888523 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.888468    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:10:55.905870 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.905812    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d5c99f576-2g548" podStartSLOduration=2.31885818 podStartE2EDuration="6.905795267s" totalImagesPullingTime="4.586937087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:55.903411432 +0000 UTC m=+102.011723026" watchObservedRunningTime="2026-07-07 14:10:55.905795267 +0000 UTC m=+102.014106860"
Jul 07 14:10:55.930207 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.930147    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq" podStartSLOduration=100.930101648 podStartE2EDuration="1m40.930101648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:55.928459741 +0000 UTC m=+102.036771336" watchObservedRunningTime="2026-07-07 14:10:55.930101648 +0000 UTC m=+102.038413241"
Jul 07 14:10:55.955325 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:10:55.955268    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6587794879-vmd5w" podStartSLOduration=2.35186169 podStartE2EDuration="6.955250428s" totalImagesPullingTime="4.603388738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:55.954762774 +0000 UTC m=+102.063074368" watchObservedRunningTime="2026-07-07 14:10:55.955250428 +0000 UTC m=+102.063562000"
Jul 07 14:11:00.649804 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.649761    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r7qg8"]
Jul 07 14:11:00.653163 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.653134    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.663691 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.663668    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 14:11:00.663954 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.663935    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 14:11:00.664640 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.664613    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cf95v\""
Jul 07 14:11:00.664768 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.664672    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 14:11:00.665769 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.665744    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 14:11:00.717003 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.716943    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.717003 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.716976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52656e2f-b141-4ea4-88a0-4ec26954c9a3-root\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.717104 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.717004    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j894\" (UniqueName: \"kubernetes.io/projected/52656e2f-b141-4ea4-88a0-4ec26954c9a3-kube-api-access-6j894\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.717104 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.717083    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.717212 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.717153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-wtmp\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.717212 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.717189    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-textfile\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.717212 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.717206    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52656e2f-b141-4ea4-88a0-4ec26954c9a3-metrics-client-ca\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.717324 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.717246    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-tls\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.717324 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.717280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52656e2f-b141-4ea4-88a0-4ec26954c9a3-sys\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.818597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52656e2f-b141-4ea4-88a0-4ec26954c9a3-root\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.818761 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j894\" (UniqueName: \"kubernetes.io/projected/52656e2f-b141-4ea4-88a0-4ec26954c9a3-kube-api-access-6j894\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.818761 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.818761 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818666    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52656e2f-b141-4ea4-88a0-4ec26954c9a3-root\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.818870 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-wtmp\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.818909 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-textfile\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.818956 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52656e2f-b141-4ea4-88a0-4ec26954c9a3-metrics-client-ca\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.818956 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-tls\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.819051 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818972    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-wtmp\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.819051 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.818974    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52656e2f-b141-4ea4-88a0-4ec26954c9a3-sys\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.819171 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.819058    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.819171 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:11:00.819075    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 14:11:00.819171 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.819060    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52656e2f-b141-4ea4-88a0-4ec26954c9a3-sys\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.819171 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:11:00.819158    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-tls podName:52656e2f-b141-4ea4-88a0-4ec26954c9a3 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:01.319137685 +0000 UTC m=+107.427449272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-tls") pod "node-exporter-r7qg8" (UID: "52656e2f-b141-4ea4-88a0-4ec26954c9a3") : secret "node-exporter-tls" not found
Jul 07 14:11:00.819364 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.819301    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.819364 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.819302    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-textfile\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.819502 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.819485    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52656e2f-b141-4ea4-88a0-4ec26954c9a3-metrics-client-ca\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.821443 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.821423    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.834902 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.834867    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j894\" (UniqueName: \"kubernetes.io/projected/52656e2f-b141-4ea4-88a0-4ec26954c9a3-kube-api-access-6j894\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:00.911303 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.911198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2kvq8" event={"ID":"27acbc3b-ba1e-411b-a893-a78af377f3ad","Type":"ContainerStarted","Data":"818ced0c244a402c6be7491250f3ba66e4caee1febba9e03bf5615fc1d8fc0d7"}
Jul 07 14:11:00.911303 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.911259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2kvq8" event={"ID":"27acbc3b-ba1e-411b-a893-a78af377f3ad","Type":"ContainerStarted","Data":"5f37a6ca1eb6343659956030049d103cba059a2c58dc08b1abcc279c4db9b126"}
Jul 07 14:11:00.911303 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.911294    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:11:00.913124 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.913083    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-45gjt" event={"ID":"6aadd26c-2770-473f-8987-aed718ce85d1","Type":"ContainerStarted","Data":"0056a02b11033bd88fd46923d537249acfab9259219fd5e4a51d4157f64d8d28"}
Jul 07 14:11:00.913243 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.913147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-45gjt" event={"ID":"6aadd26c-2770-473f-8987-aed718ce85d1","Type":"ContainerStarted","Data":"05b75874dd1d5ec661bd649747bbb8c7015e713aee93502a74a488bd6423f3d1"}
Jul 07 14:11:00.947750 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:00.947691    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2kvq8" podStartSLOduration=67.535289919 podStartE2EDuration="1m11.94766584s" totalImagesPullingTime="4.412375921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:00.946170052 +0000 UTC m=+107.054481647" watchObservedRunningTime="2026-07-07 14:11:00.94766584 +0000 UTC m=+107.055977433"
Jul 07 14:11:01.324973 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:01.324934    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-tls\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:01.327317 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:01.327288    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52656e2f-b141-4ea4-88a0-4ec26954c9a3-node-exporter-tls\") pod \"node-exporter-r7qg8\" (UID: \"52656e2f-b141-4ea4-88a0-4ec26954c9a3\") " pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:01.562586 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:01.562544    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r7qg8"
Jul 07 14:11:01.576265 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:11:01.576181    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52656e2f_b141_4ea4_88a0_4ec26954c9a3.slice/crio-804f6b12e47067d01df3ed90566c6716ca9b287ad88d6c65b6ae72039d0c838e WatchSource:0}: Error finding container 804f6b12e47067d01df3ed90566c6716ca9b287ad88d6c65b6ae72039d0c838e: Status 404 returned error can't find the container with id 804f6b12e47067d01df3ed90566c6716ca9b287ad88d6c65b6ae72039d0c838e
Jul 07 14:11:01.921692 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:01.921644    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r7qg8" event={"ID":"52656e2f-b141-4ea4-88a0-4ec26954c9a3","Type":"ContainerStarted","Data":"804f6b12e47067d01df3ed90566c6716ca9b287ad88d6c65b6ae72039d0c838e"}
Jul 07 14:11:03.931717 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:03.931683    2569 generic.go:356] "Generic (PLEG): container finished" podID="52656e2f-b141-4ea4-88a0-4ec26954c9a3" containerID="bb734f91d87581eca4653d2ad4f789e5145e09ecd7e03a15c4afc7db320663e9" exitCode=0
Jul 07 14:11:03.932244 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:03.931770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r7qg8" event={"ID":"52656e2f-b141-4ea4-88a0-4ec26954c9a3","Type":"ContainerDied","Data":"bb734f91d87581eca4653d2ad4f789e5145e09ecd7e03a15c4afc7db320663e9"}
Jul 07 14:11:03.937068 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:03.937033    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-45gjt" event={"ID":"6aadd26c-2770-473f-8987-aed718ce85d1","Type":"ContainerStarted","Data":"c0e5836b6674cb11fc997521d5931014acc99f1a28849eb5215e5ac7dc89f10c"}
Jul 07 14:11:04.943336 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:04.943286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r7qg8" event={"ID":"52656e2f-b141-4ea4-88a0-4ec26954c9a3","Type":"ContainerStarted","Data":"f457de930e86e92aeae0bcfaa18d01fa9d330ef5693a5e3dafecad34cdf50dff"}
Jul 07 14:11:04.943753 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:04.943364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r7qg8" event={"ID":"52656e2f-b141-4ea4-88a0-4ec26954c9a3","Type":"ContainerStarted","Data":"ed07fcb57b1cd8b369f85d909c2f1cb077dc0ae3b0e2866fe6e44d63ef6bf279"}
Jul 07 14:11:04.973561 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:04.973504    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r7qg8" podStartSLOduration=2.777561727 podStartE2EDuration="4.973486558s" totalImagesPullingTime="2.127656542s" totalInitContainerRuntime="68.268289ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:04.973307583 +0000 UTC m=+111.081619179" watchObservedRunningTime="2026-07-07 14:11:04.973486558 +0000 UTC m=+111.081798151"
Jul 07 14:11:04.973723 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:04.973613    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-45gjt" podStartSLOduration=34.898403426 podStartE2EDuration="42.973608174s" totalImagesPullingTime="8.075204748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:04.107917584 +0000 UTC m=+110.216229188" watchObservedRunningTime="2026-07-07 14:11:04.973608174 +0000 UTC m=+111.081919767"
Jul 07 14:11:06.439949 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:06.439915    2569 scope.go:117] "RemoveContainer" containerID="e33107cd6bc948b6553dd9ef242a4cdb8d38bf0af36311730f17bc354c7f88b3"
Jul 07 14:11:06.952620 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:06.952588    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:11:06.952815 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:06.952648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" event={"ID":"806d472c-532b-49fa-a1d6-c2a047ae8c3b","Type":"ContainerStarted","Data":"2849ea3886c2b44af8e4a6a32667c0b1583f8b32bd49a535328468a9950dd9bc"}
Jul 07 14:11:06.952932 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:06.952898    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:11:06.957616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:06.957590    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-rcshd"
Jul 07 14:11:06.996268 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:06.996195    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-rcshd" podStartSLOduration=47.628450845 podStartE2EDuration="52.996171704s" totalImagesPullingTime="5.367720859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:06.994878057 +0000 UTC m=+113.103189651" watchObservedRunningTime="2026-07-07 14:11:06.996171704 +0000 UTC m=+113.104483299"
Jul 07 14:11:10.924364 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:10.924327    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2kvq8"
Jul 07 14:11:13.750615 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:13.750578    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f47d4797f-k8jfq"]
Jul 07 14:11:13.755186 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:13.755160    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:11:24.227735 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:24.227696    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:11:24.230205 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:24.230176    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298379e-f6c7-44eb-b922-f393e007c4af-metrics-certs\") pod \"network-metrics-daemon-8jm6t\" (UID: \"4298379e-f6c7-44eb-b922-f393e007c4af\") " pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:11:24.256282 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:24.256251    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-66bkv\""
Jul 07 14:11:24.264077 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:24.264043    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jm6t"
Jul 07 14:11:24.414529 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:24.414495    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8jm6t"]
Jul 07 14:11:24.417834 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:11:24.417797    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4298379e_f6c7_44eb_b922_f393e007c4af.slice/crio-b2daad91e7e0705e7a3c71e28fd78ee7cfb9edd42837faeb596bf36112590353 WatchSource:0}: Error finding container b2daad91e7e0705e7a3c71e28fd78ee7cfb9edd42837faeb596bf36112590353: Status 404 returned error can't find the container with id b2daad91e7e0705e7a3c71e28fd78ee7cfb9edd42837faeb596bf36112590353
Jul 07 14:11:25.019538 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:25.019502    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jm6t" event={"ID":"4298379e-f6c7-44eb-b922-f393e007c4af","Type":"ContainerStarted","Data":"b2daad91e7e0705e7a3c71e28fd78ee7cfb9edd42837faeb596bf36112590353"}
Jul 07 14:11:26.025216 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:26.025178    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jm6t" event={"ID":"4298379e-f6c7-44eb-b922-f393e007c4af","Type":"ContainerStarted","Data":"f2ca841c62ae499cfc0123a73ee1c1f73526de2ff51a5451a1cc2092ce4ce3d8"}
Jul 07 14:11:27.031271 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:27.031227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jm6t" event={"ID":"4298379e-f6c7-44eb-b922-f393e007c4af","Type":"ContainerStarted","Data":"341daffc57b2f2582dc62e9aa7c0f134c4fc6d17fe6ab48eb449d30e76657228"}
Jul 07 14:11:27.051729 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:27.051671    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8jm6t" podStartSLOduration=131.632751081 podStartE2EDuration="2m13.051654098s" totalImagesPullingTime="1.418903017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:27.051180899 +0000 UTC m=+133.159492492" watchObservedRunningTime="2026-07-07 14:11:27.051654098 +0000 UTC m=+133.159965691"
Jul 07 14:11:32.051473 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:32.051388    2569 generic.go:356] "Generic (PLEG): container finished" podID="5af4992e-4e6f-4445-938c-657fd6c085ab" containerID="44dfb7d644fde15fd176afd4ea16268f5c2b33985e0ce73dd1641576fcc8aca5" exitCode=0
Jul 07 14:11:32.051473 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:32.051437    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh" event={"ID":"5af4992e-4e6f-4445-938c-657fd6c085ab","Type":"ContainerDied","Data":"44dfb7d644fde15fd176afd4ea16268f5c2b33985e0ce73dd1641576fcc8aca5"}
Jul 07 14:11:32.051890 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:32.051755    2569 scope.go:117] "RemoveContainer" containerID="44dfb7d644fde15fd176afd4ea16268f5c2b33985e0ce73dd1641576fcc8aca5"
Jul 07 14:11:33.064001 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:33.063959    2569 generic.go:356] "Generic (PLEG): container finished" podID="ba2eb9cf-295b-4d9f-aee2-28ffa025e24c" containerID="c701a99b0e028f76365be71141d3e2a5e8f2cb30faf38f25d777eb8a1745170e" exitCode=0
Jul 07 14:11:33.064509 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:33.064003    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj" event={"ID":"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c","Type":"ContainerDied","Data":"c701a99b0e028f76365be71141d3e2a5e8f2cb30faf38f25d777eb8a1745170e"}
Jul 07 14:11:33.064509 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:33.064397    2569 scope.go:117] "RemoveContainer" containerID="c701a99b0e028f76365be71141d3e2a5e8f2cb30faf38f25d777eb8a1745170e"
Jul 07 14:11:33.066239 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:33.066212    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-9ksfh" event={"ID":"5af4992e-4e6f-4445-938c-657fd6c085ab","Type":"ContainerStarted","Data":"7f5b342c2e2da3f771ab4e0613a5d681b8d230ecc3b9480b9ba7e14a61fb142a"}
Jul 07 14:11:34.072355 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:34.072315    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p4txj" event={"ID":"ba2eb9cf-295b-4d9f-aee2-28ffa025e24c","Type":"ContainerStarted","Data":"369d878e27e99fe5382f0b7ac2f7c723dc47939fa4f2518998013d431d85d0e7"}
Jul 07 14:11:38.771285 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:38.771240    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq" podUID="6b91a946-afd8-446e-af80-613ed4826f31" containerName="registry" containerID="cri-o://cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18" gracePeriod=30
Jul 07 14:11:39.037531 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.037499    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:11:39.095096 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.095040    2569 generic.go:356] "Generic (PLEG): container finished" podID="6b91a946-afd8-446e-af80-613ed4826f31" containerID="cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18" exitCode=0
Jul 07 14:11:39.095306 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.095104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq" event={"ID":"6b91a946-afd8-446e-af80-613ed4826f31","Type":"ContainerDied","Data":"cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18"}
Jul 07 14:11:39.095306 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.095203    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq" event={"ID":"6b91a946-afd8-446e-af80-613ed4826f31","Type":"ContainerDied","Data":"90f9d0d1ad2cfacb037611aa6832ee14cd79c48c5857af7fa946f872b7081e5a"}
Jul 07 14:11:39.095306 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.095244    2569 scope.go:117] "RemoveContainer" containerID="cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18"
Jul 07 14:11:39.095475 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.095454    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f47d4797f-k8jfq"
Jul 07 14:11:39.105693 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.105665    2569 scope.go:117] "RemoveContainer" containerID="cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18"
Jul 07 14:11:39.106077 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:11:39.106049    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18\": container with ID starting with cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18 not found: ID does not exist" containerID="cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18"
Jul 07 14:11:39.106165 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.106087    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18"} err="failed to get container status \"cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18\": rpc error: code = NotFound desc = could not find container \"cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18\": container with ID starting with cb9ecc2b82430d20d6e519d682624ae97507aef6d448555b7433aac46fb59b18 not found: ID does not exist"
Jul 07 14:11:39.142816 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.142777    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-registry-certificates\") pod \"6b91a946-afd8-446e-af80-613ed4826f31\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") "
Jul 07 14:11:39.143021 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.142830    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-image-registry-private-configuration\") pod \"6b91a946-afd8-446e-af80-613ed4826f31\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") "
Jul 07 14:11:39.143021 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.142873    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-bound-sa-token\") pod \"6b91a946-afd8-446e-af80-613ed4826f31\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") "
Jul 07 14:11:39.143021 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.142898    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-installation-pull-secrets\") pod \"6b91a946-afd8-446e-af80-613ed4826f31\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") "
Jul 07 14:11:39.143021 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.142918    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") pod \"6b91a946-afd8-446e-af80-613ed4826f31\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") "
Jul 07 14:11:39.143021 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.142948    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-trusted-ca\") pod \"6b91a946-afd8-446e-af80-613ed4826f31\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") "
Jul 07 14:11:39.143021 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.142980    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b91a946-afd8-446e-af80-613ed4826f31-ca-trust-extracted\") pod \"6b91a946-afd8-446e-af80-613ed4826f31\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") "
Jul 07 14:11:39.143021 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.143019    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcbv5\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-kube-api-access-wcbv5\") pod \"6b91a946-afd8-446e-af80-613ed4826f31\" (UID: \"6b91a946-afd8-446e-af80-613ed4826f31\") "
Jul 07 14:11:39.143417 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.143377    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6b91a946-afd8-446e-af80-613ed4826f31" (UID: "6b91a946-afd8-446e-af80-613ed4826f31"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:11:39.143476 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.143427    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6b91a946-afd8-446e-af80-613ed4826f31" (UID: "6b91a946-afd8-446e-af80-613ed4826f31"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:11:39.145670 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.145628    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-kube-api-access-wcbv5" (OuterVolumeSpecName: "kube-api-access-wcbv5") pod "6b91a946-afd8-446e-af80-613ed4826f31" (UID: "6b91a946-afd8-446e-af80-613ed4826f31"). InnerVolumeSpecName "kube-api-access-wcbv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:11:39.145813 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.145710    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6b91a946-afd8-446e-af80-613ed4826f31" (UID: "6b91a946-afd8-446e-af80-613ed4826f31"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:11:39.145813 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.145745    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6b91a946-afd8-446e-af80-613ed4826f31" (UID: "6b91a946-afd8-446e-af80-613ed4826f31"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:11:39.145813 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.145787    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6b91a946-afd8-446e-af80-613ed4826f31" (UID: "6b91a946-afd8-446e-af80-613ed4826f31"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:11:39.145930 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.145908    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6b91a946-afd8-446e-af80-613ed4826f31" (UID: "6b91a946-afd8-446e-af80-613ed4826f31"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:11:39.152457 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.152418    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b91a946-afd8-446e-af80-613ed4826f31-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6b91a946-afd8-446e-af80-613ed4826f31" (UID: "6b91a946-afd8-446e-af80-613ed4826f31"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:11:39.244462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.244423    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-registry-certificates\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:11:39.244462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.244460    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-image-registry-private-configuration\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:11:39.244462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.244472    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-bound-sa-token\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:11:39.244693 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.244481    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b91a946-afd8-446e-af80-613ed4826f31-installation-pull-secrets\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:11:39.244693 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.244489    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-registry-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:11:39.244693 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.244498    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b91a946-afd8-446e-af80-613ed4826f31-trusted-ca\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:11:39.244693 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.244505    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b91a946-afd8-446e-af80-613ed4826f31-ca-trust-extracted\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:11:39.244693 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.244513    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcbv5\" (UniqueName: \"kubernetes.io/projected/6b91a946-afd8-446e-af80-613ed4826f31-kube-api-access-wcbv5\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:11:39.428125 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.428070    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f47d4797f-k8jfq"]
Jul 07 14:11:39.435757 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:39.435724    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f47d4797f-k8jfq"]
Jul 07 14:11:40.440545 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:40.440510    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b91a946-afd8-446e-af80-613ed4826f31" path="/var/lib/kubelet/pods/6b91a946-afd8-446e-af80-613ed4826f31/volumes"
Jul 07 14:11:42.108366 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:42.108333    2569 generic.go:356] "Generic (PLEG): container finished" podID="85998ef5-484f-4de6-9dc5-7ded0e7317cd" containerID="978d2690dae7d5cfd85378ff075bb1142bddda0b56e6991c56dc847e56fee2e7" exitCode=0
Jul 07 14:11:42.108765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:42.108406    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j" event={"ID":"85998ef5-484f-4de6-9dc5-7ded0e7317cd","Type":"ContainerDied","Data":"978d2690dae7d5cfd85378ff075bb1142bddda0b56e6991c56dc847e56fee2e7"}
Jul 07 14:11:42.108765 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:42.108741    2569 scope.go:117] "RemoveContainer" containerID="978d2690dae7d5cfd85378ff075bb1142bddda0b56e6991c56dc847e56fee2e7"
Jul 07 14:11:43.113586 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:11:43.113547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-k8t8j" event={"ID":"85998ef5-484f-4de6-9dc5-7ded0e7317cd","Type":"ContainerStarted","Data":"96c9486e104452a614962cdbd487700901c1339f2df9efa53f391fb7e9c00653"}
Jul 07 14:14:14.320796 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:14:14.320766    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:14:14.326178 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:14:14.326148    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:14:14.330865 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:14:14.330834    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 14:15:27.436517 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.436427    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-txkqg"]
Jul 07 14:15:27.437133 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.436839    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b91a946-afd8-446e-af80-613ed4826f31" containerName="registry"
Jul 07 14:15:27.437133 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.436857    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b91a946-afd8-446e-af80-613ed4826f31" containerName="registry"
Jul 07 14:15:27.437133 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.436923    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b91a946-afd8-446e-af80-613ed4826f31" containerName="registry"
Jul 07 14:15:27.441153 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.440931    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:27.444273 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.444241    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hmn8c\""
Jul 07 14:15:27.444423 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.444241    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 07 14:15:27.445048 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.445031    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 07 14:15:27.445187 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.445046    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 07 14:15:27.445187 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.445054    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 07 14:15:27.449606 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.449576    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-txkqg"]
Jul 07 14:15:27.544381 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.544341    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/91e300a8-1bcc-44ed-9996-250f8191e5e6-certificates\") pod \"keda-admission-5fbd5c4644-txkqg\" (UID: \"91e300a8-1bcc-44ed-9996-250f8191e5e6\") " pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:27.544580 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.544388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khc98\" (UniqueName: \"kubernetes.io/projected/91e300a8-1bcc-44ed-9996-250f8191e5e6-kube-api-access-khc98\") pod \"keda-admission-5fbd5c4644-txkqg\" (UID: \"91e300a8-1bcc-44ed-9996-250f8191e5e6\") " pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:27.644980 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.644921    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/91e300a8-1bcc-44ed-9996-250f8191e5e6-certificates\") pod \"keda-admission-5fbd5c4644-txkqg\" (UID: \"91e300a8-1bcc-44ed-9996-250f8191e5e6\") " pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:27.644980 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.644981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khc98\" (UniqueName: \"kubernetes.io/projected/91e300a8-1bcc-44ed-9996-250f8191e5e6-kube-api-access-khc98\") pod \"keda-admission-5fbd5c4644-txkqg\" (UID: \"91e300a8-1bcc-44ed-9996-250f8191e5e6\") " pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:27.645302 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:15:27.645101    2569 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 07 14:15:27.645302 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:15:27.645165    2569 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-txkqg: secret "keda-admission-webhooks-certs" not found
Jul 07 14:15:27.645302 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:15:27.645241    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91e300a8-1bcc-44ed-9996-250f8191e5e6-certificates podName:91e300a8-1bcc-44ed-9996-250f8191e5e6 nodeName:}" failed. No retries permitted until 2026-07-07 14:15:28.145218174 +0000 UTC m=+374.253529747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/91e300a8-1bcc-44ed-9996-250f8191e5e6-certificates") pod "keda-admission-5fbd5c4644-txkqg" (UID: "91e300a8-1bcc-44ed-9996-250f8191e5e6") : secret "keda-admission-webhooks-certs" not found
Jul 07 14:15:27.657668 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:27.657637    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khc98\" (UniqueName: \"kubernetes.io/projected/91e300a8-1bcc-44ed-9996-250f8191e5e6-kube-api-access-khc98\") pod \"keda-admission-5fbd5c4644-txkqg\" (UID: \"91e300a8-1bcc-44ed-9996-250f8191e5e6\") " pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:28.148970 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:28.148930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/91e300a8-1bcc-44ed-9996-250f8191e5e6-certificates\") pod \"keda-admission-5fbd5c4644-txkqg\" (UID: \"91e300a8-1bcc-44ed-9996-250f8191e5e6\") " pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:28.151530 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:28.151502    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/91e300a8-1bcc-44ed-9996-250f8191e5e6-certificates\") pod \"keda-admission-5fbd5c4644-txkqg\" (UID: \"91e300a8-1bcc-44ed-9996-250f8191e5e6\") " pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:28.354505 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:28.354458    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:28.495753 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:28.495720    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-txkqg"]
Jul 07 14:15:28.499188 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:15:28.499157    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e300a8_1bcc_44ed_9996_250f8191e5e6.slice/crio-5a0d58d9c3f39d4a9a15f7772bf1aa6c9ba576a63bfe1e9e6bc790acc45f7773 WatchSource:0}: Error finding container 5a0d58d9c3f39d4a9a15f7772bf1aa6c9ba576a63bfe1e9e6bc790acc45f7773: Status 404 returned error can't find the container with id 5a0d58d9c3f39d4a9a15f7772bf1aa6c9ba576a63bfe1e9e6bc790acc45f7773
Jul 07 14:15:28.500480 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:28.500459    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:15:28.947289 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:28.947236    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-txkqg" event={"ID":"91e300a8-1bcc-44ed-9996-250f8191e5e6","Type":"ContainerStarted","Data":"5a0d58d9c3f39d4a9a15f7772bf1aa6c9ba576a63bfe1e9e6bc790acc45f7773"}
Jul 07 14:15:30.956530 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:30.956491    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-txkqg" event={"ID":"91e300a8-1bcc-44ed-9996-250f8191e5e6","Type":"ContainerStarted","Data":"54fd3b5e5fe824e98bed3c4fa2ee382454e1e36505f2327f0238b9787b884405"}
Jul 07 14:15:30.956927 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:30.956626    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:15:30.976242 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:30.976192    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-txkqg" podStartSLOduration=1.837863325 podStartE2EDuration="3.976172174s" totalImagesPullingTime="2.138308849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:15:30.975304527 +0000 UTC m=+377.083616112" watchObservedRunningTime="2026-07-07 14:15:30.976172174 +0000 UTC m=+377.084483768"
Jul 07 14:15:51.963315 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:15:51.963276    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-txkqg"
Jul 07 14:16:35.282402 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.282364    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-lkq96"]
Jul 07 14:16:35.285890 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.285867    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:35.288313 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.288288    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 14:16:35.288455 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.288336    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ddcns\""
Jul 07 14:16:35.288941 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.288925    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 14:16:35.289001 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.288983    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 07 14:16:35.294739 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.294712    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-lkq96"]
Jul 07 14:16:35.307396 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.307360    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-6wtqp"]
Jul 07 14:16:35.310694 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.310664    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:16:35.313242 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.313217    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 14:16:35.313367 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.313264    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zrcbk\""
Jul 07 14:16:35.320741 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.320708    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6wtqp"]
Jul 07 14:16:35.379805 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.379759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppzcq\" (UniqueName: \"kubernetes.io/projected/11a9ed90-2d3f-4ccd-a4ae-ec5824677d33-kube-api-access-ppzcq\") pod \"llmisvc-controller-manager-67d95fdc48-lkq96\" (UID: \"11a9ed90-2d3f-4ccd-a4ae-ec5824677d33\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:35.379992 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.379832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/067647ef-0ba5-4d3b-a9fb-800a34f49e00-data\") pod \"seaweedfs-64568bcd49-6wtqp\" (UID: \"067647ef-0ba5-4d3b-a9fb-800a34f49e00\") " pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:16:35.379992 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.379905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11a9ed90-2d3f-4ccd-a4ae-ec5824677d33-cert\") pod \"llmisvc-controller-manager-67d95fdc48-lkq96\" (UID: \"11a9ed90-2d3f-4ccd-a4ae-ec5824677d33\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:35.379992 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.379930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd47t\" (UniqueName: \"kubernetes.io/projected/067647ef-0ba5-4d3b-a9fb-800a34f49e00-kube-api-access-kd47t\") pod \"seaweedfs-64568bcd49-6wtqp\" (UID: \"067647ef-0ba5-4d3b-a9fb-800a34f49e00\") " pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:16:35.480398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.480352    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppzcq\" (UniqueName: \"kubernetes.io/projected/11a9ed90-2d3f-4ccd-a4ae-ec5824677d33-kube-api-access-ppzcq\") pod \"llmisvc-controller-manager-67d95fdc48-lkq96\" (UID: \"11a9ed90-2d3f-4ccd-a4ae-ec5824677d33\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:35.480398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.480402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/067647ef-0ba5-4d3b-a9fb-800a34f49e00-data\") pod \"seaweedfs-64568bcd49-6wtqp\" (UID: \"067647ef-0ba5-4d3b-a9fb-800a34f49e00\") " pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:16:35.480653 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.480466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11a9ed90-2d3f-4ccd-a4ae-ec5824677d33-cert\") pod \"llmisvc-controller-manager-67d95fdc48-lkq96\" (UID: \"11a9ed90-2d3f-4ccd-a4ae-ec5824677d33\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:35.480653 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.480495    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd47t\" (UniqueName: \"kubernetes.io/projected/067647ef-0ba5-4d3b-a9fb-800a34f49e00-kube-api-access-kd47t\") pod \"seaweedfs-64568bcd49-6wtqp\" (UID: \"067647ef-0ba5-4d3b-a9fb-800a34f49e00\") " pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:16:35.480653 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:16:35.480617    2569 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 07 14:16:35.480803 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:16:35.480683    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11a9ed90-2d3f-4ccd-a4ae-ec5824677d33-cert podName:11a9ed90-2d3f-4ccd-a4ae-ec5824677d33 nodeName:}" failed. No retries permitted until 2026-07-07 14:16:35.980667096 +0000 UTC m=+442.088978668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/11a9ed90-2d3f-4ccd-a4ae-ec5824677d33-cert") pod "llmisvc-controller-manager-67d95fdc48-lkq96" (UID: "11a9ed90-2d3f-4ccd-a4ae-ec5824677d33") : secret "llmisvc-webhook-server-cert" not found
Jul 07 14:16:35.480849 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.480813    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/067647ef-0ba5-4d3b-a9fb-800a34f49e00-data\") pod \"seaweedfs-64568bcd49-6wtqp\" (UID: \"067647ef-0ba5-4d3b-a9fb-800a34f49e00\") " pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:16:35.490830 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.490796    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd47t\" (UniqueName: \"kubernetes.io/projected/067647ef-0ba5-4d3b-a9fb-800a34f49e00-kube-api-access-kd47t\") pod \"seaweedfs-64568bcd49-6wtqp\" (UID: \"067647ef-0ba5-4d3b-a9fb-800a34f49e00\") " pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:16:35.490944 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.490837    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppzcq\" (UniqueName: \"kubernetes.io/projected/11a9ed90-2d3f-4ccd-a4ae-ec5824677d33-kube-api-access-ppzcq\") pod \"llmisvc-controller-manager-67d95fdc48-lkq96\" (UID: \"11a9ed90-2d3f-4ccd-a4ae-ec5824677d33\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:35.625653 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.625558    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:16:35.766778 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.766737    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6wtqp"]
Jul 07 14:16:35.770531 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:16:35.770493    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067647ef_0ba5_4d3b_a9fb_800a34f49e00.slice/crio-2cc02fed436401d2b8b7dd71a665bc4fc1f90d0e858e21bb068384aa78f93685 WatchSource:0}: Error finding container 2cc02fed436401d2b8b7dd71a665bc4fc1f90d0e858e21bb068384aa78f93685: Status 404 returned error can't find the container with id 2cc02fed436401d2b8b7dd71a665bc4fc1f90d0e858e21bb068384aa78f93685
Jul 07 14:16:35.985041 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.985002    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11a9ed90-2d3f-4ccd-a4ae-ec5824677d33-cert\") pod \"llmisvc-controller-manager-67d95fdc48-lkq96\" (UID: \"11a9ed90-2d3f-4ccd-a4ae-ec5824677d33\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:35.987443 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:35.987421    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11a9ed90-2d3f-4ccd-a4ae-ec5824677d33-cert\") pod \"llmisvc-controller-manager-67d95fdc48-lkq96\" (UID: \"11a9ed90-2d3f-4ccd-a4ae-ec5824677d33\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:36.197419 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:36.197373    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:36.203140 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:36.203093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6wtqp" event={"ID":"067647ef-0ba5-4d3b-a9fb-800a34f49e00","Type":"ContainerStarted","Data":"2cc02fed436401d2b8b7dd71a665bc4fc1f90d0e858e21bb068384aa78f93685"}
Jul 07 14:16:36.336059 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:36.336022    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-lkq96"]
Jul 07 14:16:36.339558 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:16:36.339529    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod11a9ed90_2d3f_4ccd_a4ae_ec5824677d33.slice/crio-2ac946984278b97f03b6dc9d6b619d3046e5abc0d594940604ab968d0dacc14d WatchSource:0}: Error finding container 2ac946984278b97f03b6dc9d6b619d3046e5abc0d594940604ab968d0dacc14d: Status 404 returned error can't find the container with id 2ac946984278b97f03b6dc9d6b619d3046e5abc0d594940604ab968d0dacc14d
Jul 07 14:16:37.208946 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:37.208908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96" event={"ID":"11a9ed90-2d3f-4ccd-a4ae-ec5824677d33","Type":"ContainerStarted","Data":"2ac946984278b97f03b6dc9d6b619d3046e5abc0d594940604ab968d0dacc14d"}
Jul 07 14:16:39.218929 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:39.218883    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96" event={"ID":"11a9ed90-2d3f-4ccd-a4ae-ec5824677d33","Type":"ContainerStarted","Data":"35d889ad95e56546b7dad93821b3fa605be64220ac99be5efd0cd6e7bf12d864"}
Jul 07 14:16:39.219352 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:39.218968    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:16:39.237383 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:39.237326    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96" podStartSLOduration=2.045795543 podStartE2EDuration="4.237308633s" totalImagesPullingTime="2.19151309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:16:39.23625506 +0000 UTC m=+445.344566658" watchObservedRunningTime="2026-07-07 14:16:39.237308633 +0000 UTC m=+445.345620227"
Jul 07 14:16:41.233125 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:41.233071    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6wtqp" event={"ID":"067647ef-0ba5-4d3b-a9fb-800a34f49e00","Type":"ContainerStarted","Data":"5f760b44c09a4d37b0e1d97f8ea54c4f82fef29f796d7b0913aa801b292a5665"}
Jul 07 14:16:41.233561 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:41.233186    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:16:41.254515 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:41.254443    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-6wtqp" podStartSLOduration=1.214357298 podStartE2EDuration="6.254419282s" totalImagesPullingTime="5.040061984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:16:41.252468112 +0000 UTC m=+447.360779707" watchObservedRunningTime="2026-07-07 14:16:41.254419282 +0000 UTC m=+447.362730878"
Jul 07 14:16:47.239499 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:16:47.239466    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-6wtqp"
Jul 07 14:17:10.225387 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:10.225355    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-lkq96"
Jul 07 14:17:45.880785 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:45.880749    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-ckpm2"]
Jul 07 14:17:45.883837 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:45.883818    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:45.887340 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:45.887316    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-58785\""
Jul 07 14:17:45.887486 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:45.887357    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 07 14:17:45.896395 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:45.896358    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-ckpm2"]
Jul 07 14:17:45.968034 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:45.967983    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbcx\" (UniqueName: \"kubernetes.io/projected/6dc1e5f4-d961-4fd7-9145-773705c9f344-kube-api-access-kzbcx\") pod \"model-serving-api-fd65d7d6-ckpm2\" (UID: \"6dc1e5f4-d961-4fd7-9145-773705c9f344\") " pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:45.968034 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:45.968035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dc1e5f4-d961-4fd7-9145-773705c9f344-tls-certs\") pod \"model-serving-api-fd65d7d6-ckpm2\" (UID: \"6dc1e5f4-d961-4fd7-9145-773705c9f344\") " pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:46.068830 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:46.068778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbcx\" (UniqueName: \"kubernetes.io/projected/6dc1e5f4-d961-4fd7-9145-773705c9f344-kube-api-access-kzbcx\") pod \"model-serving-api-fd65d7d6-ckpm2\" (UID: \"6dc1e5f4-d961-4fd7-9145-773705c9f344\") " pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:46.068830 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:46.068833    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dc1e5f4-d961-4fd7-9145-773705c9f344-tls-certs\") pod \"model-serving-api-fd65d7d6-ckpm2\" (UID: \"6dc1e5f4-d961-4fd7-9145-773705c9f344\") " pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:46.069050 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:17:46.068964    2569 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 07 14:17:46.069050 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:17:46.069038    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dc1e5f4-d961-4fd7-9145-773705c9f344-tls-certs podName:6dc1e5f4-d961-4fd7-9145-773705c9f344 nodeName:}" failed. No retries permitted until 2026-07-07 14:17:46.569022393 +0000 UTC m=+512.677333977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6dc1e5f4-d961-4fd7-9145-773705c9f344-tls-certs") pod "model-serving-api-fd65d7d6-ckpm2" (UID: "6dc1e5f4-d961-4fd7-9145-773705c9f344") : secret "model-serving-api-tls" not found
Jul 07 14:17:46.083454 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:46.083419    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbcx\" (UniqueName: \"kubernetes.io/projected/6dc1e5f4-d961-4fd7-9145-773705c9f344-kube-api-access-kzbcx\") pod \"model-serving-api-fd65d7d6-ckpm2\" (UID: \"6dc1e5f4-d961-4fd7-9145-773705c9f344\") " pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:46.573844 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:46.573799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dc1e5f4-d961-4fd7-9145-773705c9f344-tls-certs\") pod \"model-serving-api-fd65d7d6-ckpm2\" (UID: \"6dc1e5f4-d961-4fd7-9145-773705c9f344\") " pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:46.576291 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:46.576264    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dc1e5f4-d961-4fd7-9145-773705c9f344-tls-certs\") pod \"model-serving-api-fd65d7d6-ckpm2\" (UID: \"6dc1e5f4-d961-4fd7-9145-773705c9f344\") " pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:46.795029 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:46.794989    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:46.935349 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:46.935315    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-ckpm2"]
Jul 07 14:17:46.939284 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:17:46.939249    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc1e5f4_d961_4fd7_9145_773705c9f344.slice/crio-8b443fe87876699ea5c8ec833c523bd10977f3dfa713e08afb0a80811456477e WatchSource:0}: Error finding container 8b443fe87876699ea5c8ec833c523bd10977f3dfa713e08afb0a80811456477e: Status 404 returned error can't find the container with id 8b443fe87876699ea5c8ec833c523bd10977f3dfa713e08afb0a80811456477e
Jul 07 14:17:47.475844 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:47.475803    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-ckpm2" event={"ID":"6dc1e5f4-d961-4fd7-9145-773705c9f344","Type":"ContainerStarted","Data":"8b443fe87876699ea5c8ec833c523bd10977f3dfa713e08afb0a80811456477e"}
Jul 07 14:17:49.485171 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:49.485105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-ckpm2" event={"ID":"6dc1e5f4-d961-4fd7-9145-773705c9f344","Type":"ContainerStarted","Data":"a3fb311bd89816e5c713bd2ccf9f3b13666616a51aaf4a49df118a9d09a7bf70"}
Jul 07 14:17:49.485630 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:49.485193    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:17:49.506965 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:17:49.506875    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-ckpm2" podStartSLOduration=2.076541694 podStartE2EDuration="4.506857582s" totalImagesPullingTime="2.430315888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:17:49.505637817 +0000 UTC m=+515.613949414" watchObservedRunningTime="2026-07-07 14:17:49.506857582 +0000 UTC m=+515.615169176"
Jul 07 14:18:00.493409 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:18:00.493379    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-ckpm2"
Jul 07 14:19:14.350134 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:19:14.350079    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:19:14.363543 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:19:14.363514    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:20:34.587536 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.587496    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"]
Jul 07 14:20:34.590929 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.590910    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.595246 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.595217    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:20:34.595449 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.595274    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 14:20:34.595568 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.595546    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 14:20:34.596256 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.596227    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 07 14:20:34.596256 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.596248    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rswgx\""
Jul 07 14:20:34.596407 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.596263    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 07 14:20:34.596407 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.596290    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:20:34.605387 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.605356    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"]
Jul 07 14:20:34.655188 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.655104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.655388 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.655249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.655388 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.655294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42j8\" (UniqueName: \"kubernetes.io/projected/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-api-access-r42j8\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.655388 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.655316    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.655388 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.655358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.756619 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.756576    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.756619 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.756627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.756855 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.756673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.756855 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.756702    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42j8\" (UniqueName: \"kubernetes.io/projected/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-api-access-r42j8\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.756855 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.756728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.757156 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.757103    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.757363 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.757345    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.757436 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.757417    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.759194 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.759173    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.767411 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.767383    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42j8\" (UniqueName: \"kubernetes.io/projected/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-api-access-r42j8\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:34.901398 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:34.901300    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:20:35.045249 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:35.045222    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"]
Jul 07 14:20:35.047845 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:20:35.047816    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b2aa00_ae0c_4b62_a95f_743a97da5d78.slice/crio-304eef2f5f2feac16716a217147cc0c728919546402568fa0293e99f83feb526 WatchSource:0}: Error finding container 304eef2f5f2feac16716a217147cc0c728919546402568fa0293e99f83feb526: Status 404 returned error can't find the container with id 304eef2f5f2feac16716a217147cc0c728919546402568fa0293e99f83feb526
Jul 07 14:20:35.049813 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:35.049796    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:20:35.112442 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:35.112405    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerStarted","Data":"304eef2f5f2feac16716a217147cc0c728919546402568fa0293e99f83feb526"}
Jul 07 14:20:39.137924 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:39.137884    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerStarted","Data":"bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc"}
Jul 07 14:20:43.156413 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:43.156375    2569 generic.go:356] "Generic (PLEG): container finished" podID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerID="bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc" exitCode=0
Jul 07 14:20:43.156805 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:43.156446    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerDied","Data":"bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc"}
Jul 07 14:20:59.235055 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:20:59.235015    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerStarted","Data":"c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef"}
Jul 07 14:21:02.250325 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:02.250285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerStarted","Data":"d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14"}
Jul 07 14:21:05.265963 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:05.265916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerStarted","Data":"399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4"}
Jul 07 14:21:05.266409 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:05.266026    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:21:05.266409 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:05.266055    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:21:05.267707 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:05.267664    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:21:05.329736 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:05.329670    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podStartSLOduration=3.098195125 podStartE2EDuration="31.3296494s" totalImagesPullingTime="24.599663436s" totalInitContainerRuntime="3.631790839s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:20:34 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:21:05.32829936 +0000 UTC m=+711.436610954" watchObservedRunningTime="2026-07-07 14:21:05.3296494 +0000 UTC m=+711.437960997"
Jul 07 14:21:06.271129 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:06.271078    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:21:06.271719 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:06.271254    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:21:06.272519 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:06.272484    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:21:06.275604 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:06.275580    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:21:07.275763 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:07.275721    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:21:07.276181 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:07.276100    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:21:08.279341 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:08.279301    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:21:08.279735 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:08.279551    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:21:18.279922 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:18.279874    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:21:18.280399 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:18.280371    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:21:28.279832 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:28.279776    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:21:28.280389 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:28.280362    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:21:38.279994 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:38.279947    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:21:38.280565 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:38.280473    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:21:48.280162 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:48.280081    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:21:48.280613 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:48.280519    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:21:58.280239 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:58.280186    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:21:58.280666 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:21:58.280583    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:22:08.279361 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:08.279301    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:22:08.279852 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:08.279796    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:22:18.280010 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:18.279973    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:22:18.280495 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:18.280230    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:22:29.971228 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:29.971184    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"]
Jul 07 14:22:29.971836 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:29.971473    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" containerID="cri-o://c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef" gracePeriod=30
Jul 07 14:22:29.971836 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:29.971504    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" containerID="cri-o://399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4" gracePeriod=30
Jul 07 14:22:29.971836 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:29.971573    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy" containerID="cri-o://d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14" gracePeriod=30
Jul 07 14:22:29.975149 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:29.975099    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"]
Jul 07 14:22:29.979041 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:29.979014    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:29.982033 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:29.982012    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 07 14:22:29.982161 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:29.982049    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 07 14:22:29.995240 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:29.995201    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"]
Jul 07 14:22:30.028600 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.028562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd41e939-70bc-44ca-ae94-c10fdb289557-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.028778 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.028615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.028778 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.028727    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.028855 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.028778    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq72j\" (UniqueName: \"kubernetes.io/projected/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-api-access-qq72j\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.028855 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.028801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd41e939-70bc-44ca-ae94-c10fdb289557-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.130239 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.130175    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.130424 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.130257    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq72j\" (UniqueName: \"kubernetes.io/projected/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-api-access-qq72j\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.130424 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.130291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd41e939-70bc-44ca-ae94-c10fdb289557-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.130499 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.130421    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd41e939-70bc-44ca-ae94-c10fdb289557-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.130499 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.130484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.130734 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.130712    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd41e939-70bc-44ca-ae94-c10fdb289557-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.130979 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.130958    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.131188 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.131165    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.133203 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.133181    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd41e939-70bc-44ca-ae94-c10fdb289557-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.143276 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.143242    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq72j\" (UniqueName: \"kubernetes.io/projected/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-api-access-qq72j\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.295254 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.295158    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:30.457718 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.453723    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"]
Jul 07 14:22:30.615654 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.615569    2569 generic.go:356] "Generic (PLEG): container finished" podID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerID="d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14" exitCode=2
Jul 07 14:22:30.615654 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.615641    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerDied","Data":"d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14"}
Jul 07 14:22:30.617328 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.617295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerStarted","Data":"02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780"}
Jul 07 14:22:30.617460 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:30.617361    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerStarted","Data":"b857605d26bda90b5aab541df09ef096a1f5701349c745206bef82ade4689b2d"}
Jul 07 14:22:31.271818 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:31.271776    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 07 14:22:34.635895 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:34.635857    2569 generic.go:356] "Generic (PLEG): container finished" podID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerID="02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780" exitCode=0
Jul 07 14:22:34.636385 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:34.635929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerDied","Data":"02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780"}
Jul 07 14:22:35.642799 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.642763    2569 generic.go:356] "Generic (PLEG): container finished" podID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerID="c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef" exitCode=0
Jul 07 14:22:35.643309 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.642831    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerDied","Data":"c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef"}
Jul 07 14:22:35.645263 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.645236    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerStarted","Data":"b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b"}
Jul 07 14:22:35.645414 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.645307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerStarted","Data":"794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271"}
Jul 07 14:22:35.645414 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.645344    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerStarted","Data":"fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e"}
Jul 07 14:22:35.645736 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.645711    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:35.645815 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.645779    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:35.645815 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.645813    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:35.647293 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.647259    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:22:35.647882 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.647853    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:22:35.674853 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:35.674791    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podStartSLOduration=3.03977932 podStartE2EDuration="6.674771741s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.634992421s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:22:35.674477291 +0000 UTC m=+801.782788879" watchObservedRunningTime="2026-07-07 14:22:35.674771741 +0000 UTC m=+801.783083334"
Jul 07 14:22:36.271774 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:36.271728    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 07 14:22:36.651999 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:36.651864    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:22:36.652402 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:36.652317    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:22:38.279876 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:38.279827    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:22:38.280388 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:38.280163    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:22:41.271927 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:41.271875    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 07 14:22:41.272383 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:41.272030    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:22:41.656673 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:41.656576    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:22:41.657288 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:41.657256    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:22:41.657587 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:41.657564    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:22:46.271569 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:46.271521    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 07 14:22:48.280009 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:48.279961    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:22:48.280488 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:48.280327    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:22:51.272121 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:51.272075    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 07 14:22:51.657763 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:51.657650    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:22:51.658100 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:51.658059    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:22:56.272406 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:56.272360    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 07 14:22:58.279468 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:58.279409    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 07 14:22:58.279881 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:58.279564    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:22:58.279881 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:58.279786    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:22:58.279881 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:22:58.279876    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:23:00.145228 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.145181    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:23:00.292544 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.292453    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kserve-provision-location\") pod \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") "
Jul 07 14:23:00.292544 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.292529    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-rbac-proxy-sar-config\") pod \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") "
Jul 07 14:23:00.292743 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.292556    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r42j8\" (UniqueName: \"kubernetes.io/projected/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-api-access-r42j8\") pod \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") "
Jul 07 14:23:00.292743 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.292581    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-cabundle-cert\") pod \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") "
Jul 07 14:23:00.292743 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.292608    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-proxy-tls\") pod \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\" (UID: \"f3b2aa00-ae0c-4b62-a95f-743a97da5d78\") "
Jul 07 14:23:00.292941 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.292847    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3b2aa00-ae0c-4b62-a95f-743a97da5d78" (UID: "f3b2aa00-ae0c-4b62-a95f-743a97da5d78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:23:00.292989 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.292964    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3b2aa00-ae0c-4b62-a95f-743a97da5d78" (UID: "f3b2aa00-ae0c-4b62-a95f-743a97da5d78"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:23:00.293032 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.293009    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3b2aa00-ae0c-4b62-a95f-743a97da5d78" (UID: "f3b2aa00-ae0c-4b62-a95f-743a97da5d78"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:23:00.294963 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.294930    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-api-access-r42j8" (OuterVolumeSpecName: "kube-api-access-r42j8") pod "f3b2aa00-ae0c-4b62-a95f-743a97da5d78" (UID: "f3b2aa00-ae0c-4b62-a95f-743a97da5d78"). InnerVolumeSpecName "kube-api-access-r42j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:23:00.294963 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.294930    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3b2aa00-ae0c-4b62-a95f-743a97da5d78" (UID: "f3b2aa00-ae0c-4b62-a95f-743a97da5d78"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:23:00.393181 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.393141    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-proxy-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:23:00.393181 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.393177    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kserve-provision-location\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:23:00.393181 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.393188    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:23:00.393415 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.393197    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r42j8\" (UniqueName: \"kubernetes.io/projected/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-kube-api-access-r42j8\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:23:00.393415 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.393206    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2aa00-ae0c-4b62-a95f-743a97da5d78-cabundle-cert\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:23:00.755587 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.755556    2569 generic.go:356] "Generic (PLEG): container finished" podID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerID="399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4" exitCode=0
Jul 07 14:23:00.755776 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.755620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerDied","Data":"399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4"}
Jul 07 14:23:00.755776 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.755643    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"
Jul 07 14:23:00.755776 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.755711    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n" event={"ID":"f3b2aa00-ae0c-4b62-a95f-743a97da5d78","Type":"ContainerDied","Data":"304eef2f5f2feac16716a217147cc0c728919546402568fa0293e99f83feb526"}
Jul 07 14:23:00.755776 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.755757    2569 scope.go:117] "RemoveContainer" containerID="399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4"
Jul 07 14:23:00.764648 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.764626    2569 scope.go:117] "RemoveContainer" containerID="d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14"
Jul 07 14:23:00.773006 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.772982    2569 scope.go:117] "RemoveContainer" containerID="c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef"
Jul 07 14:23:00.783478 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.783405    2569 scope.go:117] "RemoveContainer" containerID="bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc"
Jul 07 14:23:00.785809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.785785    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"]
Jul 07 14:23:00.787191 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.787168    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-92g7n"]
Jul 07 14:23:00.792837 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.792818    2569 scope.go:117] "RemoveContainer" containerID="399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4"
Jul 07 14:23:00.793169 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:23:00.793143    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4\": container with ID starting with 399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4 not found: ID does not exist" containerID="399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4"
Jul 07 14:23:00.793326 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.793174    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4"} err="failed to get container status \"399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4\": rpc error: code = NotFound desc = could not find container \"399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4\": container with ID starting with 399750ab69915ab72b675ec48141d7aede5544f1a8b36fc2481acaa5b48231e4 not found: ID does not exist"
Jul 07 14:23:00.793326 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.793198    2569 scope.go:117] "RemoveContainer" containerID="d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14"
Jul 07 14:23:00.793474 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:23:00.793453    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14\": container with ID starting with d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14 not found: ID does not exist" containerID="d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14"
Jul 07 14:23:00.793562 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.793475    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14"} err="failed to get container status \"d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14\": rpc error: code = NotFound desc = could not find container \"d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14\": container with ID starting with d59b55e8e6ee540cd971bd0e54772a072ed76fab368df7708665263e52029d14 not found: ID does not exist"
Jul 07 14:23:00.793562 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.793489    2569 scope.go:117] "RemoveContainer" containerID="c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef"
Jul 07 14:23:00.793775 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:23:00.793757    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef\": container with ID starting with c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef not found: ID does not exist" containerID="c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef"
Jul 07 14:23:00.793819 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.793777    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef"} err="failed to get container status \"c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef\": rpc error: code = NotFound desc = could not find container \"c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef\": container with ID starting with c94f9eff32d1f06ef54db58e413e08412725db2b7e671194758d58cabbb3bdef not found: ID does not exist"
Jul 07 14:23:00.793819 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.793789    2569 scope.go:117] "RemoveContainer" containerID="bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc"
Jul 07 14:23:00.794020 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:23:00.794005    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc\": container with ID starting with bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc not found: ID does not exist" containerID="bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc"
Jul 07 14:23:00.794060 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:00.794022    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc"} err="failed to get container status \"bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc\": rpc error: code = NotFound desc = could not find container \"bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc\": container with ID starting with bd4e1f0ad336c706a0548caa0d61c8c9010af80e2181ed48e2d9c9309f71cacc not found: ID does not exist"
Jul 07 14:23:01.657315 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:01.657271    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:23:01.657812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:01.657745    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:23:02.440742 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:02.440694    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" path="/var/lib/kubelet/pods/f3b2aa00-ae0c-4b62-a95f-743a97da5d78/volumes"
Jul 07 14:23:11.657614 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:11.657566    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:23:11.658100 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:11.657977    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:23:21.657841 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:21.657792    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:23:21.658375 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:21.658352    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:23:31.657339 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:31.657288    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:23:31.657836 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:31.657816    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:23:41.657462 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:41.657414    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:23:41.657995 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:41.657970    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:23:51.658363 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:51.658329    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:23:51.658877 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:23:51.658422    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:24:05.126878 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.126840    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"]
Jul 07 14:24:05.127377 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127320    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container"
Jul 07 14:24:05.127377 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127339    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container"
Jul 07 14:24:05.127377 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127354    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent"
Jul 07 14:24:05.127377 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127363    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent"
Jul 07 14:24:05.127590 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127390    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy"
Jul 07 14:24:05.127590 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127400    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy"
Jul 07 14:24:05.127590 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127415    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="storage-initializer"
Jul 07 14:24:05.127590 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127424    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="storage-initializer"
Jul 07 14:24:05.127590 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127504    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kserve-container"
Jul 07 14:24:05.127590 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127516    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="agent"
Jul 07 14:24:05.127590 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.127526    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b2aa00-ae0c-4b62-a95f-743a97da5d78" containerName="kube-rbac-proxy"
Jul 07 14:24:05.130946 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.130915    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.135428 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.135390    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 07 14:24:05.135632 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.135611    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 07 14:24:05.145498 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.145465    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"]
Jul 07 14:24:05.160836 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.160796    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"]
Jul 07 14:24:05.161970 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.161440    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" containerID="cri-o://b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b" gracePeriod=30
Jul 07 14:24:05.161970 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.161550    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy" containerID="cri-o://794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271" gracePeriod=30
Jul 07 14:24:05.162930 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.162731    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" containerID="cri-o://fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e" gracePeriod=30
Jul 07 14:24:05.216325 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.216283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-xx8rd\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.216325 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.216331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-xx8rd\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.216525 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.216349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zltqx\" (UniqueName: \"kubernetes.io/projected/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-api-access-zltqx\") pod \"message-dumper-predictor-6575bcfdfd-xx8rd\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.317850 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.317804    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-xx8rd\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.317850 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.317855    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-xx8rd\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.318180 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.317873    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zltqx\" (UniqueName: \"kubernetes.io/projected/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-api-access-zltqx\") pod \"message-dumper-predictor-6575bcfdfd-xx8rd\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.318604 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.318577    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-xx8rd\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.320332 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.320310    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-xx8rd\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.328236 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.328208    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zltqx\" (UniqueName: \"kubernetes.io/projected/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-api-access-zltqx\") pod \"message-dumper-predictor-6575bcfdfd-xx8rd\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.470271 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.470226    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:05.613700 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:05.613670    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"]
Jul 07 14:24:05.615638 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:24:05.615612    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89ebb4e_4eae_4d6c_8da0_b099b36d3b73.slice/crio-6a486d753446a08f616407c7fc3d1bfab94bac9db7c23781bfbbae7d005701ac WatchSource:0}: Error finding container 6a486d753446a08f616407c7fc3d1bfab94bac9db7c23781bfbbae7d005701ac: Status 404 returned error can't find the container with id 6a486d753446a08f616407c7fc3d1bfab94bac9db7c23781bfbbae7d005701ac
Jul 07 14:24:06.019537 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:06.019502    2569 generic.go:356] "Generic (PLEG): container finished" podID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerID="794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271" exitCode=2
Jul 07 14:24:06.019723 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:06.019586    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerDied","Data":"794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271"}
Jul 07 14:24:06.020802 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:06.020780    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd" event={"ID":"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73","Type":"ContainerStarted","Data":"6a486d753446a08f616407c7fc3d1bfab94bac9db7c23781bfbbae7d005701ac"}
Jul 07 14:24:06.652598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:06.652547    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 07 14:24:07.027002 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:07.026970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd" event={"ID":"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73","Type":"ContainerStarted","Data":"e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48"}
Jul 07 14:24:08.032761 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:08.032721    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd" event={"ID":"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73","Type":"ContainerStarted","Data":"15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b"}
Jul 07 14:24:08.033210 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:08.032873    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:08.033210 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:08.032920    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:08.034705 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:08.034677    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:08.056324 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:08.056271    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd" podStartSLOduration=1.767474454 podStartE2EDuration="3.056252969s" totalImagesPullingTime="1.288778515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:08.054331392 +0000 UTC m=+894.162642989" watchObservedRunningTime="2026-07-07 14:24:08.056252969 +0000 UTC m=+894.164564563"
Jul 07 14:24:11.048480 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:11.048445    2569 generic.go:356] "Generic (PLEG): container finished" podID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerID="fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e" exitCode=0
Jul 07 14:24:11.048879 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:11.048521    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerDied","Data":"fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e"}
Jul 07 14:24:11.652022 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:11.651976    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 07 14:24:11.657396 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:11.657364    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:24:11.657744 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:11.657715    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:14.383012 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:14.382981    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:24:14.387662 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:14.387636    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:24:15.047135 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:15.047083    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:24:16.651815 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:16.651775    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 07 14:24:16.652218 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:16.651916    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:24:21.652578 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:21.652527    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 07 14:24:21.658073 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:21.658034    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:24:21.658443 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:21.658411    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:25.160300 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.160258    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"]
Jul 07 14:24:25.164411 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.164379    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.167012 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.166981    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 07 14:24:25.167012 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.166982    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 07 14:24:25.179085 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.179050    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"]
Jul 07 14:24:25.281919 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.281876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9nbg\" (UniqueName: \"kubernetes.io/projected/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-api-access-g9nbg\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.281919 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.281925    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.282186 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.281974    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.282186 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.281997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.282186 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.282015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94529fa6-8c8f-4312-a94d-5ea59319c9f1-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.382639 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.382597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9nbg\" (UniqueName: \"kubernetes.io/projected/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-api-access-g9nbg\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.382639 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.382645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.382911 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.382683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.382911 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.382707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.382911 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.382727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94529fa6-8c8f-4312-a94d-5ea59319c9f1-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.383264 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.383231    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.383496 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.383472    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.383561 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.383526    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.385521 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.385498    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94529fa6-8c8f-4312-a94d-5ea59319c9f1-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.395031 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.394983    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9nbg\" (UniqueName: \"kubernetes.io/projected/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-api-access-g9nbg\") pod \"isvc-logger-predictor-599b89d7c-s8n6c\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.475025 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.474988    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:25.629881 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:25.629851    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"]
Jul 07 14:24:25.634938 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:24:25.634905    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94529fa6_8c8f_4312_a94d_5ea59319c9f1.slice/crio-c12b37f094926950c538ff2e6dd5f245985b6cf6aebf4e470fd2c8aec59d1684 WatchSource:0}: Error finding container c12b37f094926950c538ff2e6dd5f245985b6cf6aebf4e470fd2c8aec59d1684: Status 404 returned error can't find the container with id c12b37f094926950c538ff2e6dd5f245985b6cf6aebf4e470fd2c8aec59d1684
Jul 07 14:24:26.112378 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:26.112336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerStarted","Data":"a1710c86535aa577416924eb3550e64b4653d995aac601e6b6aa91003028b2af"}
Jul 07 14:24:26.112555 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:26.112431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerStarted","Data":"c12b37f094926950c538ff2e6dd5f245985b6cf6aebf4e470fd2c8aec59d1684"}
Jul 07 14:24:26.652900 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:26.652852    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 07 14:24:30.133149 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:30.133087    2569 generic.go:356] "Generic (PLEG): container finished" podID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerID="a1710c86535aa577416924eb3550e64b4653d995aac601e6b6aa91003028b2af" exitCode=0
Jul 07 14:24:30.133566 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:30.133142    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerDied","Data":"a1710c86535aa577416924eb3550e64b4653d995aac601e6b6aa91003028b2af"}
Jul 07 14:24:31.139927 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.139892    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerStarted","Data":"a5364beab671624cde93dff879a4540bd097576a429e66b74cc02df3e0b61400"}
Jul 07 14:24:31.140366 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.139962    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerStarted","Data":"2a4f9f79b9c39252bd0fb67dc46e5b5f28aa8f045777d2193e22ea056f6c2cad"}
Jul 07 14:24:31.140366 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.139986    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerStarted","Data":"20c0ef3c0c3e14f7068c1be70ceeae515cd8f755c72f4db9fb8562a13eb2a482"}
Jul 07 14:24:31.140366 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.140290    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:31.140512 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.140432    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:31.141810 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.141783    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:24:31.168595 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.168521    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podStartSLOduration=2.311152077 podStartE2EDuration="6.168502223s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.857350146s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:31.166947517 +0000 UTC m=+917.275259113" watchObservedRunningTime="2026-07-07 14:24:31.168502223 +0000 UTC m=+917.276813816"
Jul 07 14:24:31.652319 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.652267    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 07 14:24:31.657612 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.657578    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:5000: connect: connection refused"
Jul 07 14:24:31.657738 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.657710    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:24:31.658016 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.657990    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:31.658106 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:31.658076    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:24:32.144911 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:32.144877    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:32.145402 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:32.144989    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:24:32.145928 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:32.145907    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:33.149953 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:33.149900    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:24:33.150420 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:33.150227    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:35.316381 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.316352    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:24:35.366677 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.366639    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd41e939-70bc-44ca-ae94-c10fdb289557-kserve-provision-location\") pod \"cd41e939-70bc-44ca-ae94-c10fdb289557\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") "
Jul 07 14:24:35.366845 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.366700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq72j\" (UniqueName: \"kubernetes.io/projected/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-api-access-qq72j\") pod \"cd41e939-70bc-44ca-ae94-c10fdb289557\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") "
Jul 07 14:24:35.366845 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.366739    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-cabundle-cert\") pod \"cd41e939-70bc-44ca-ae94-c10fdb289557\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") "
Jul 07 14:24:35.366845 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.366818    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-rbac-proxy-sar-config\") pod \"cd41e939-70bc-44ca-ae94-c10fdb289557\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") "
Jul 07 14:24:35.367126 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.367079    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd41e939-70bc-44ca-ae94-c10fdb289557-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd41e939-70bc-44ca-ae94-c10fdb289557" (UID: "cd41e939-70bc-44ca-ae94-c10fdb289557"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:24:35.367244 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.367219    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd41e939-70bc-44ca-ae94-c10fdb289557" (UID: "cd41e939-70bc-44ca-ae94-c10fdb289557"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:24:35.367303 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.367223    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd41e939-70bc-44ca-ae94-c10fdb289557" (UID: "cd41e939-70bc-44ca-ae94-c10fdb289557"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:24:35.368876 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.368848    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-api-access-qq72j" (OuterVolumeSpecName: "kube-api-access-qq72j") pod "cd41e939-70bc-44ca-ae94-c10fdb289557" (UID: "cd41e939-70bc-44ca-ae94-c10fdb289557"). InnerVolumeSpecName "kube-api-access-qq72j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:24:35.467514 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.467425    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd41e939-70bc-44ca-ae94-c10fdb289557-proxy-tls\") pod \"cd41e939-70bc-44ca-ae94-c10fdb289557\" (UID: \"cd41e939-70bc-44ca-ae94-c10fdb289557\") "
Jul 07 14:24:35.467650 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.467573    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:24:35.467650 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.467584    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd41e939-70bc-44ca-ae94-c10fdb289557-kserve-provision-location\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:24:35.467650 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.467593    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq72j\" (UniqueName: \"kubernetes.io/projected/cd41e939-70bc-44ca-ae94-c10fdb289557-kube-api-access-qq72j\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:24:35.467650 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.467601    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd41e939-70bc-44ca-ae94-c10fdb289557-cabundle-cert\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:24:35.469510 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.469486    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd41e939-70bc-44ca-ae94-c10fdb289557-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd41e939-70bc-44ca-ae94-c10fdb289557" (UID: "cd41e939-70bc-44ca-ae94-c10fdb289557"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:24:35.568467 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:35.568431    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd41e939-70bc-44ca-ae94-c10fdb289557-proxy-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:24:36.164926 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.164891    2569 generic.go:356] "Generic (PLEG): container finished" podID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerID="b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b" exitCode=0
Jul 07 14:24:36.165106 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.164938    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerDied","Data":"b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b"}
Jul 07 14:24:36.165106 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.164951    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"
Jul 07 14:24:36.165106 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.164990    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q" event={"ID":"cd41e939-70bc-44ca-ae94-c10fdb289557","Type":"ContainerDied","Data":"b857605d26bda90b5aab541df09ef096a1f5701349c745206bef82ade4689b2d"}
Jul 07 14:24:36.165106 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.165021    2569 scope.go:117] "RemoveContainer" containerID="b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b"
Jul 07 14:24:36.173985 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.173946    2569 scope.go:117] "RemoveContainer" containerID="794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271"
Jul 07 14:24:36.181572 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.181552    2569 scope.go:117] "RemoveContainer" containerID="fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e"
Jul 07 14:24:36.190275 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.190191    2569 scope.go:117] "RemoveContainer" containerID="02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780"
Jul 07 14:24:36.192719 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.192695    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"]
Jul 07 14:24:36.198739 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.198719    2569 scope.go:117] "RemoveContainer" containerID="b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b"
Jul 07 14:24:36.199056 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:24:36.199037    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b\": container with ID starting with b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b not found: ID does not exist" containerID="b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b"
Jul 07 14:24:36.199177 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.199068    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b"} err="failed to get container status \"b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b\": rpc error: code = NotFound desc = could not find container \"b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b\": container with ID starting with b80db920de3a5dea0b4a698041039811b4b083754bd828e3d6b2f673df38db5b not found: ID does not exist"
Jul 07 14:24:36.199177 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.199089    2569 scope.go:117] "RemoveContainer" containerID="794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271"
Jul 07 14:24:36.199409 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.199392    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-q6w6q"]
Jul 07 14:24:36.199472 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:24:36.199455    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271\": container with ID starting with 794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271 not found: ID does not exist" containerID="794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271"
Jul 07 14:24:36.199508 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.199479    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271"} err="failed to get container status \"794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271\": rpc error: code = NotFound desc = could not find container \"794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271\": container with ID starting with 794d5bfce826f8d91adc3830578e6f24bf150efb5be4f5238175d46bd58d8271 not found: ID does not exist"
Jul 07 14:24:36.199508 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.199494    2569 scope.go:117] "RemoveContainer" containerID="fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e"
Jul 07 14:24:36.199803 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:24:36.199789    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e\": container with ID starting with fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e not found: ID does not exist" containerID="fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e"
Jul 07 14:24:36.199865 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.199805    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e"} err="failed to get container status \"fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e\": rpc error: code = NotFound desc = could not find container \"fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e\": container with ID starting with fefa2f572c36a2c93b8e84a1d8a9a8ed0544825df4d991027909c717db02273e not found: ID does not exist"
Jul 07 14:24:36.199865 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.199817    2569 scope.go:117] "RemoveContainer" containerID="02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780"
Jul 07 14:24:36.200020 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:24:36.200003    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780\": container with ID starting with 02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780 not found: ID does not exist" containerID="02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780"
Jul 07 14:24:36.200071 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.200024    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780"} err="failed to get container status \"02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780\": rpc error: code = NotFound desc = could not find container \"02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780\": container with ID starting with 02de49573f8774f50e935c6f29acfa1dba1e4b0c8297e3c78d338d6e07868780 not found: ID does not exist"
Jul 07 14:24:36.440533 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:36.440455    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" path="/var/lib/kubelet/pods/cd41e939-70bc-44ca-ae94-c10fdb289557/volumes"
Jul 07 14:24:38.154309 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:38.154281    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:24:38.154899 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:38.154861    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:24:38.155227 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:38.155204    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:48.155395 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:48.155354    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:24:48.155919 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:48.155895    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:58.155204 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:58.155155    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:24:58.155736 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:24:58.155656    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:08.155364 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:08.155315    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:25:08.155857 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:08.155751    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:18.157874 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:18.157819    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:25:18.159506 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:18.159468    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:28.155523 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:28.155470    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:25:28.156028 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:28.156003    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:38.155867 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:38.155757    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:25:38.158343 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:38.156268    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:48.156041 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:48.156003    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:25:48.156476 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:25:48.156231    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:26:00.184254 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.184223    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-xx8rd_e89ebb4e-4eae-4d6c-8da0-b099b36d3b73/kserve-container/0.log"
Jul 07 14:26:00.625679 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.625603    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"]
Jul 07 14:26:00.625893 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.625848    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" containerID="cri-o://20c0ef3c0c3e14f7068c1be70ceeae515cd8f755c72f4db9fb8562a13eb2a482" gracePeriod=30
Jul 07 14:26:00.625981 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.625947    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy" containerID="cri-o://2a4f9f79b9c39252bd0fb67dc46e5b5f28aa8f045777d2193e22ea056f6c2cad" gracePeriod=30
Jul 07 14:26:00.626038 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.625869    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" containerID="cri-o://a5364beab671624cde93dff879a4540bd097576a429e66b74cc02df3e0b61400" gracePeriod=30
Jul 07 14:26:00.629906 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.629879    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"]
Jul 07 14:26:00.630240 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630224    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy"
Jul 07 14:26:00.630240 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630239    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy"
Jul 07 14:26:00.630393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630250    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container"
Jul 07 14:26:00.630393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630257    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container"
Jul 07 14:26:00.630393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630266    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="storage-initializer"
Jul 07 14:26:00.630393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630272    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="storage-initializer"
Jul 07 14:26:00.630393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630287    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent"
Jul 07 14:26:00.630393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630293    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent"
Jul 07 14:26:00.630393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630346    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="agent"
Jul 07 14:26:00.630393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630352    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kube-rbac-proxy"
Jul 07 14:26:00.630393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.630360    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd41e939-70bc-44ca-ae94-c10fdb289557" containerName="kserve-container"
Jul 07 14:26:00.633678 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.633657    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.636404 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.636384    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 07 14:26:00.636508 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.636455    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 07 14:26:00.648916 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.648887    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"]
Jul 07 14:26:00.691937 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.691906    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"]
Jul 07 14:26:00.692287 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.692252    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd" podUID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerName="kserve-container" containerID="cri-o://e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48" gracePeriod=30
Jul 07 14:26:00.692287 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.692268    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd" podUID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerName="kube-rbac-proxy" containerID="cri-o://15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b" gracePeriod=30
Jul 07 14:26:00.767238 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.767200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.767434 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.767254    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdqj\" (UniqueName: \"kubernetes.io/projected/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-api-access-2gdqj\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.767434 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.767333    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c443757-aebb-4999-9826-cdb6f82f27d4-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.767434 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.767404    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.767583 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.767488    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c443757-aebb-4999-9826-cdb6f82f27d4-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.868618 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.868586    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c443757-aebb-4999-9826-cdb6f82f27d4-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.868771 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.868636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.868771 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.868674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdqj\" (UniqueName: \"kubernetes.io/projected/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-api-access-2gdqj\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.868771 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.868709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c443757-aebb-4999-9826-cdb6f82f27d4-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.868771 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.868746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.869236 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.869204    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c443757-aebb-4999-9826-cdb6f82f27d4-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.869515 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.869494    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.869650 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.869626    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.871402 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.871372    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c443757-aebb-4999-9826-cdb6f82f27d4-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.880158 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.880076    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdqj\" (UniqueName: \"kubernetes.io/projected/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-api-access-2gdqj\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:00.938507 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.938481    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:26:00.949826 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:00.949793    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:26:01.070639 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.070602    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-rbac-proxy-sar-config\") pod \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") "
Jul 07 14:26:01.070791 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.070664    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-proxy-tls\") pod \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") "
Jul 07 14:26:01.070791 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.070732    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zltqx\" (UniqueName: \"kubernetes.io/projected/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-api-access-zltqx\") pod \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\" (UID: \"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73\") "
Jul 07 14:26:01.071063 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.071026    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" (UID: "e89ebb4e-4eae-4d6c-8da0-b099b36d3b73"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:26:01.072724 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.072700    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" (UID: "e89ebb4e-4eae-4d6c-8da0-b099b36d3b73"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:26:01.072982 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.072962    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-api-access-zltqx" (OuterVolumeSpecName: "kube-api-access-zltqx") pod "e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" (UID: "e89ebb4e-4eae-4d6c-8da0-b099b36d3b73"). InnerVolumeSpecName "kube-api-access-zltqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:26:01.101231 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.101194    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"]
Jul 07 14:26:01.104356 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:26:01.104328    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c443757_aebb_4999_9826_cdb6f82f27d4.slice/crio-fc09075ce7b1a5a905d4f317f320cab3c58bcd4b3bcdebaf89747d824777e337 WatchSource:0}: Error finding container fc09075ce7b1a5a905d4f317f320cab3c58bcd4b3bcdebaf89747d824777e337: Status 404 returned error can't find the container with id fc09075ce7b1a5a905d4f317f320cab3c58bcd4b3bcdebaf89747d824777e337
Jul 07 14:26:01.172178 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.172150    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:26:01.172178 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.172177    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-proxy-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:26:01.172178 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.172188    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zltqx\" (UniqueName: \"kubernetes.io/projected/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73-kube-api-access-zltqx\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:26:01.539170 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.539135    2569 generic.go:356] "Generic (PLEG): container finished" podID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerID="2a4f9f79b9c39252bd0fb67dc46e5b5f28aa8f045777d2193e22ea056f6c2cad" exitCode=2
Jul 07 14:26:01.539594 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.539210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerDied","Data":"2a4f9f79b9c39252bd0fb67dc46e5b5f28aa8f045777d2193e22ea056f6c2cad"}
Jul 07 14:26:01.540839 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.540820    2569 generic.go:356] "Generic (PLEG): container finished" podID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerID="15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b" exitCode=2
Jul 07 14:26:01.540839 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.540836    2569 generic.go:356] "Generic (PLEG): container finished" podID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerID="e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48" exitCode=2
Jul 07 14:26:01.541002 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.540871    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"
Jul 07 14:26:01.541002 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.540901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd" event={"ID":"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73","Type":"ContainerDied","Data":"15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b"}
Jul 07 14:26:01.541002 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.540972    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd" event={"ID":"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73","Type":"ContainerDied","Data":"e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48"}
Jul 07 14:26:01.541168 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.541005    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd" event={"ID":"e89ebb4e-4eae-4d6c-8da0-b099b36d3b73","Type":"ContainerDied","Data":"6a486d753446a08f616407c7fc3d1bfab94bac9db7c23781bfbbae7d005701ac"}
Jul 07 14:26:01.541168 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.541031    2569 scope.go:117] "RemoveContainer" containerID="15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b"
Jul 07 14:26:01.542443 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.542412    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" event={"ID":"8c443757-aebb-4999-9826-cdb6f82f27d4","Type":"ContainerStarted","Data":"5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825"}
Jul 07 14:26:01.542556 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.542471    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" event={"ID":"8c443757-aebb-4999-9826-cdb6f82f27d4","Type":"ContainerStarted","Data":"fc09075ce7b1a5a905d4f317f320cab3c58bcd4b3bcdebaf89747d824777e337"}
Jul 07 14:26:01.550871 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.550853    2569 scope.go:117] "RemoveContainer" containerID="e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48"
Jul 07 14:26:01.561641 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.561514    2569 scope.go:117] "RemoveContainer" containerID="15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b"
Jul 07 14:26:01.562047 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:26:01.562013    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b\": container with ID starting with 15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b not found: ID does not exist" containerID="15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b"
Jul 07 14:26:01.562224 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.562055    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b"} err="failed to get container status \"15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b\": rpc error: code = NotFound desc = could not find container \"15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b\": container with ID starting with 15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b not found: ID does not exist"
Jul 07 14:26:01.562224 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.562084    2569 scope.go:117] "RemoveContainer" containerID="e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48"
Jul 07 14:26:01.562553 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:26:01.562523    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48\": container with ID starting with e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48 not found: ID does not exist" containerID="e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48"
Jul 07 14:26:01.562652 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.562561    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48"} err="failed to get container status \"e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48\": rpc error: code = NotFound desc = could not find container \"e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48\": container with ID starting with e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48 not found: ID does not exist"
Jul 07 14:26:01.562652 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.562585    2569 scope.go:117] "RemoveContainer" containerID="15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b"
Jul 07 14:26:01.562989 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.562964    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b"} err="failed to get container status \"15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b\": rpc error: code = NotFound desc = could not find container \"15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b\": container with ID starting with 15e1d70fc861315a7b6e0dabda2a95d9ebc7a95f334df2c81dcb4b9379ad507b not found: ID does not exist"
Jul 07 14:26:01.562989 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.562989    2569 scope.go:117] "RemoveContainer" containerID="e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48"
Jul 07 14:26:01.563520 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.563494    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48"} err="failed to get container status \"e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48\": rpc error: code = NotFound desc = could not find container \"e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48\": container with ID starting with e4117234668339ef9cd931acf92c12bddad209a3170cb8c823db72a911dd0c48 not found: ID does not exist"
Jul 07 14:26:01.579478 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.579447    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"]
Jul 07 14:26:01.589968 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:01.589940    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-xx8rd"]
Jul 07 14:26:02.441280 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:02.441250    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" path="/var/lib/kubelet/pods/e89ebb4e-4eae-4d6c-8da0-b099b36d3b73/volumes"
Jul 07 14:26:03.149945 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:03.149904    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 14:26:05.570054 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:05.570021    2569 generic.go:356] "Generic (PLEG): container finished" podID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerID="20c0ef3c0c3e14f7068c1be70ceeae515cd8f755c72f4db9fb8562a13eb2a482" exitCode=0
Jul 07 14:26:05.570054 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:05.570064    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerDied","Data":"20c0ef3c0c3e14f7068c1be70ceeae515cd8f755c72f4db9fb8562a13eb2a482"}
Jul 07 14:26:08.150410 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:08.150364    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 14:26:08.155758 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:08.155725    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:26:08.156148 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:08.156102    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:26:13.150617 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:13.150568    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 14:26:13.151015 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:13.150736    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:26:18.150754 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:18.150706    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 14:26:18.155177 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:18.155141    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:26:18.155473 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:18.155440    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:26:23.150611 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:23.150568    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 14:26:28.150985 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:28.150941    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 14:26:28.155393 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:28.155360    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 07 14:26:28.155550 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:28.155479    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:26:28.155722 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:28.155687    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:26:28.155809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:28.155798    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:26:30.675774 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.675742    2569 generic.go:356] "Generic (PLEG): container finished" podID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerID="a5364beab671624cde93dff879a4540bd097576a429e66b74cc02df3e0b61400" exitCode=0
Jul 07 14:26:30.676236 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.675776    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerDied","Data":"a5364beab671624cde93dff879a4540bd097576a429e66b74cc02df3e0b61400"}
Jul 07 14:26:30.804175 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.804148    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:26:30.925905 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.925878    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-cabundle-cert\") pod \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") "
Jul 07 14:26:30.926081 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.925925    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9nbg\" (UniqueName: \"kubernetes.io/projected/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-api-access-g9nbg\") pod \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") "
Jul 07 14:26:30.926081 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.925946    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-rbac-proxy-sar-config\") pod \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") "
Jul 07 14:26:30.926081 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.925998    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kserve-provision-location\") pod \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") "
Jul 07 14:26:30.926081 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.926035    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94529fa6-8c8f-4312-a94d-5ea59319c9f1-proxy-tls\") pod \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\" (UID: \"94529fa6-8c8f-4312-a94d-5ea59319c9f1\") "
Jul 07 14:26:30.926418 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.926387    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94529fa6-8c8f-4312-a94d-5ea59319c9f1" (UID: "94529fa6-8c8f-4312-a94d-5ea59319c9f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:26:30.926507 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.926403    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94529fa6-8c8f-4312-a94d-5ea59319c9f1" (UID: "94529fa6-8c8f-4312-a94d-5ea59319c9f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:26:30.926507 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.926399    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94529fa6-8c8f-4312-a94d-5ea59319c9f1" (UID: "94529fa6-8c8f-4312-a94d-5ea59319c9f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:26:30.928060 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.928025    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-api-access-g9nbg" (OuterVolumeSpecName: "kube-api-access-g9nbg") pod "94529fa6-8c8f-4312-a94d-5ea59319c9f1" (UID: "94529fa6-8c8f-4312-a94d-5ea59319c9f1"). InnerVolumeSpecName "kube-api-access-g9nbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:26:30.928186 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:30.928064    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94529fa6-8c8f-4312-a94d-5ea59319c9f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94529fa6-8c8f-4312-a94d-5ea59319c9f1" (UID: "94529fa6-8c8f-4312-a94d-5ea59319c9f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:26:31.027500 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.027454    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:26:31.027500 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.027487    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kserve-provision-location\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:26:31.027500 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.027496    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94529fa6-8c8f-4312-a94d-5ea59319c9f1-proxy-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:26:31.027500 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.027507    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94529fa6-8c8f-4312-a94d-5ea59319c9f1-cabundle-cert\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:26:31.027500 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.027515    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9nbg\" (UniqueName: \"kubernetes.io/projected/94529fa6-8c8f-4312-a94d-5ea59319c9f1-kube-api-access-g9nbg\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:26:31.682288 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.682255    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"
Jul 07 14:26:31.682754 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.682254    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c" event={"ID":"94529fa6-8c8f-4312-a94d-5ea59319c9f1","Type":"ContainerDied","Data":"c12b37f094926950c538ff2e6dd5f245985b6cf6aebf4e470fd2c8aec59d1684"}
Jul 07 14:26:31.682754 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.682453    2569 scope.go:117] "RemoveContainer" containerID="a5364beab671624cde93dff879a4540bd097576a429e66b74cc02df3e0b61400"
Jul 07 14:26:31.683937 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.683913    2569 generic.go:356] "Generic (PLEG): container finished" podID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerID="5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825" exitCode=0
Jul 07 14:26:31.684018 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.683995    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" event={"ID":"8c443757-aebb-4999-9826-cdb6f82f27d4","Type":"ContainerDied","Data":"5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825"}
Jul 07 14:26:31.685763 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.685746    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:26:31.692144 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.692106    2569 scope.go:117] "RemoveContainer" containerID="2a4f9f79b9c39252bd0fb67dc46e5b5f28aa8f045777d2193e22ea056f6c2cad"
Jul 07 14:26:31.702370 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.702345    2569 scope.go:117] "RemoveContainer" containerID="20c0ef3c0c3e14f7068c1be70ceeae515cd8f755c72f4db9fb8562a13eb2a482"
Jul 07 14:26:31.710180 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.710154    2569 scope.go:117] "RemoveContainer" containerID="a1710c86535aa577416924eb3550e64b4653d995aac601e6b6aa91003028b2af"
Jul 07 14:26:31.733363 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.733332    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"]
Jul 07 14:26:31.736567 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:31.736540    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-s8n6c"]
Jul 07 14:26:32.442650 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:26:32.442599    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" path="/var/lib/kubelet/pods/94529fa6-8c8f-4312-a94d-5ea59319c9f1/volumes"
Jul 07 14:28:44.281635 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:28:44.281591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" event={"ID":"8c443757-aebb-4999-9826-cdb6f82f27d4","Type":"ContainerStarted","Data":"c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a"}
Jul 07 14:28:44.282172 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:28:44.281681    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" event={"ID":"8c443757-aebb-4999-9826-cdb6f82f27d4","Type":"ContainerStarted","Data":"e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4"}
Jul 07 14:28:44.282172 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:28:44.281791    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:28:44.332156 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:28:44.332082    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" podStartSLOduration=3.195900592 podStartE2EDuration="2m44.332062257s" totalImagesPullingTime="2m11.564225235s" totalInitContainerRuntime="29.57193643s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:28:44.331308331 +0000 UTC m=+1170.439619936" watchObservedRunningTime="2026-07-07 14:28:44.332062257 +0000 UTC m=+1170.440373850"
Jul 07 14:28:45.286130 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:28:45.286079    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:28:45.287597 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:28:45.287564    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 14:28:46.290244 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:28:46.290190    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 14:28:51.294692 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:28:51.294662    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:28:51.295402 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:28:51.295363    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 14:29:01.296279 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:01.296245    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:29:10.800714 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:10.800682    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"]
Jul 07 14:29:10.801282 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:10.800996    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kserve-container" containerID="cri-o://e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4" gracePeriod=30
Jul 07 14:29:10.801282 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:10.801056    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kube-rbac-proxy" containerID="cri-o://c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a" gracePeriod=30
Jul 07 14:29:11.290586 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.290521    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 07 14:29:11.295905 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.295865    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 14:29:11.391253 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.391219    2569 generic.go:356] "Generic (PLEG): container finished" podID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerID="c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a" exitCode=2
Jul 07 14:29:11.391416 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.391300    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" event={"ID":"8c443757-aebb-4999-9826-cdb6f82f27d4","Type":"ContainerDied","Data":"c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a"}
Jul 07 14:29:11.781841 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.781805    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:29:11.853264 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.853168    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c443757-aebb-4999-9826-cdb6f82f27d4-kserve-provision-location\") pod \"8c443757-aebb-4999-9826-cdb6f82f27d4\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") "
Jul 07 14:29:11.853264 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.853224    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c443757-aebb-4999-9826-cdb6f82f27d4-proxy-tls\") pod \"8c443757-aebb-4999-9826-cdb6f82f27d4\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") "
Jul 07 14:29:11.853264 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.853260    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gdqj\" (UniqueName: \"kubernetes.io/projected/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-api-access-2gdqj\") pod \"8c443757-aebb-4999-9826-cdb6f82f27d4\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") "
Jul 07 14:29:11.853860 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.853286    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-cabundle-cert\") pod \"8c443757-aebb-4999-9826-cdb6f82f27d4\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") "
Jul 07 14:29:11.853860 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.853322    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-rbac-proxy-sar-config\") pod \"8c443757-aebb-4999-9826-cdb6f82f27d4\" (UID: \"8c443757-aebb-4999-9826-cdb6f82f27d4\") "
Jul 07 14:29:11.853860 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.853743    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c443757-aebb-4999-9826-cdb6f82f27d4" (UID: "8c443757-aebb-4999-9826-cdb6f82f27d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:29:11.853860 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.853765    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c443757-aebb-4999-9826-cdb6f82f27d4" (UID: "8c443757-aebb-4999-9826-cdb6f82f27d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:29:11.855404 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.855377    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c443757-aebb-4999-9826-cdb6f82f27d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c443757-aebb-4999-9826-cdb6f82f27d4" (UID: "8c443757-aebb-4999-9826-cdb6f82f27d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:29:11.855533 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.855471    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-api-access-2gdqj" (OuterVolumeSpecName: "kube-api-access-2gdqj") pod "8c443757-aebb-4999-9826-cdb6f82f27d4" (UID: "8c443757-aebb-4999-9826-cdb6f82f27d4"). InnerVolumeSpecName "kube-api-access-2gdqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:29:11.857917 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.857887    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c443757-aebb-4999-9826-cdb6f82f27d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c443757-aebb-4999-9826-cdb6f82f27d4" (UID: "8c443757-aebb-4999-9826-cdb6f82f27d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:29:11.954241 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.954196    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c443757-aebb-4999-9826-cdb6f82f27d4-proxy-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:29:11.954241 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.954234    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdqj\" (UniqueName: \"kubernetes.io/projected/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-api-access-2gdqj\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:29:11.954241 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.954244    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-cabundle-cert\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:29:11.954459 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.954252    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c443757-aebb-4999-9826-cdb6f82f27d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:29:11.954459 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:11.954261    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c443757-aebb-4999-9826-cdb6f82f27d4-kserve-provision-location\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:29:12.397847 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.397808    2569 generic.go:356] "Generic (PLEG): container finished" podID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerID="e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4" exitCode=0
Jul 07 14:29:12.398054 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.397855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" event={"ID":"8c443757-aebb-4999-9826-cdb6f82f27d4","Type":"ContainerDied","Data":"e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4"}
Jul 07 14:29:12.398054 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.397873    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"
Jul 07 14:29:12.398054 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.397921    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn" event={"ID":"8c443757-aebb-4999-9826-cdb6f82f27d4","Type":"ContainerDied","Data":"fc09075ce7b1a5a905d4f317f320cab3c58bcd4b3bcdebaf89747d824777e337"}
Jul 07 14:29:12.398054 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.397948    2569 scope.go:117] "RemoveContainer" containerID="c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a"
Jul 07 14:29:12.407014 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.406995    2569 scope.go:117] "RemoveContainer" containerID="e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4"
Jul 07 14:29:12.414331 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.414310    2569 scope.go:117] "RemoveContainer" containerID="5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825"
Jul 07 14:29:12.421292 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.421276    2569 scope.go:117] "RemoveContainer" containerID="c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a"
Jul 07 14:29:12.421553 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:29:12.421531    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a\": container with ID starting with c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a not found: ID does not exist" containerID="c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a"
Jul 07 14:29:12.421599 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.421586    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a"} err="failed to get container status \"c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a\": rpc error: code = NotFound desc = could not find container \"c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a\": container with ID starting with c9d5f551e006f1035c92fc421456e69e4855f0f74ef20a6294a94f5202f06e8a not found: ID does not exist"
Jul 07 14:29:12.421645 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.421604    2569 scope.go:117] "RemoveContainer" containerID="e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4"
Jul 07 14:29:12.421890 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:29:12.421821    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4\": container with ID starting with e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4 not found: ID does not exist" containerID="e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4"
Jul 07 14:29:12.421890 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.421868    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4"} err="failed to get container status \"e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4\": rpc error: code = NotFound desc = could not find container \"e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4\": container with ID starting with e1ca95f1ee8ceb9a2dcba84188fac7a8ab16b041c234422be1d51b437e4567e4 not found: ID does not exist"
Jul 07 14:29:12.422011 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.421891    2569 scope.go:117] "RemoveContainer" containerID="5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825"
Jul 07 14:29:12.422140 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:29:12.422103    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825\": container with ID starting with 5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825 not found: ID does not exist" containerID="5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825"
Jul 07 14:29:12.422189 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.422144    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825"} err="failed to get container status \"5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825\": rpc error: code = NotFound desc = could not find container \"5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825\": container with ID starting with 5590adbc59d9cd6108665a0e680f3b842dbd58b071892d2b9fba4191dc2a5825 not found: ID does not exist"
Jul 07 14:29:12.422736 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.422719    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"]
Jul 07 14:29:12.428225 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.428204    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-2x7hn"]
Jul 07 14:29:12.440334 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:12.440315    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" path="/var/lib/kubelet/pods/8c443757-aebb-4999-9826-cdb6f82f27d4/volumes"
Jul 07 14:29:14.408253 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:14.408223    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:29:14.411211 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:14.411186    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:29:16.267239 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267202    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"]
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267484    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerName="kube-rbac-proxy"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267495    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerName="kube-rbac-proxy"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267509    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267515    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267523    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="storage-initializer"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267529    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="storage-initializer"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267537    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267543    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267552    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kserve-container"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267558    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kserve-container"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267565    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kube-rbac-proxy"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267570    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kube-rbac-proxy"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267577    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="storage-initializer"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267582    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="storage-initializer"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267590    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerName="kserve-container"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267595    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerName="kserve-container"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267602    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy"
Jul 07 14:29:16.267616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267607    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy"
Jul 07 14:29:16.268160 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267653    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kube-rbac-proxy"
Jul 07 14:29:16.268160 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267661    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerName="kserve-container"
Jul 07 14:29:16.268160 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267667    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kube-rbac-proxy"
Jul 07 14:29:16.268160 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267674    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="kserve-container"
Jul 07 14:29:16.268160 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267679    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94529fa6-8c8f-4312-a94d-5ea59319c9f1" containerName="agent"
Jul 07 14:29:16.268160 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267684    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c443757-aebb-4999-9826-cdb6f82f27d4" containerName="kserve-container"
Jul 07 14:29:16.268160 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.267690    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89ebb4e-4eae-4d6c-8da0-b099b36d3b73" containerName="kube-rbac-proxy"
Jul 07 14:29:16.291417 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.291382    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.296406 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.296378    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"]
Jul 07 14:29:16.298301 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.298272    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 14:29:16.298437 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.298287    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:29:16.298437 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.298360    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 14:29:16.299435 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.299156    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 07 14:29:16.299435 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.299211    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:29:16.299435 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.299281    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rswgx\""
Jul 07 14:29:16.299681 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.299663    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 07 14:29:16.392542 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.392501    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.392736 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.392552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.392736 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.392644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.392736 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.392696    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbq7\" (UniqueName: \"kubernetes.io/projected/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-api-access-qxbq7\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.392736 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.392721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.493424 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.493384    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbq7\" (UniqueName: \"kubernetes.io/projected/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-api-access-qxbq7\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.493424 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.493427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.493709 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.493501    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.493709 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.493524    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.493709 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.493548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.493990 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.493969    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.494305 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.494281    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.494389 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.494308    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.496212 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.496193    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.510174 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.510144    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbq7\" (UniqueName: \"kubernetes.io/projected/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-api-access-qxbq7\") pod \"isvc-autogluon-v2-predictor-67b849d456-mm4w6\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.603665 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.603564    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:16.814741 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:16.814717    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"]
Jul 07 14:29:16.817225 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:29:16.817194    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a7f32e_9f27_4859_bba4_3c2659fd1c40.slice/crio-209c7110796527bd5f015d18f031075cc0c3a6e5730a9519fb3c097f836f8ff7 WatchSource:0}: Error finding container 209c7110796527bd5f015d18f031075cc0c3a6e5730a9519fb3c097f836f8ff7: Status 404 returned error can't find the container with id 209c7110796527bd5f015d18f031075cc0c3a6e5730a9519fb3c097f836f8ff7
Jul 07 14:29:17.419587 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:17.419543    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" event={"ID":"b2a7f32e-9f27-4859-bba4-3c2659fd1c40","Type":"ContainerStarted","Data":"cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0"}
Jul 07 14:29:17.419967 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:17.419622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" event={"ID":"b2a7f32e-9f27-4859-bba4-3c2659fd1c40","Type":"ContainerStarted","Data":"209c7110796527bd5f015d18f031075cc0c3a6e5730a9519fb3c097f836f8ff7"}
Jul 07 14:29:46.541342 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:46.541307    2569 generic.go:356] "Generic (PLEG): container finished" podID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerID="cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0" exitCode=0
Jul 07 14:29:46.541754 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:46.541373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" event={"ID":"b2a7f32e-9f27-4859-bba4-3c2659fd1c40","Type":"ContainerDied","Data":"cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0"}
Jul 07 14:29:47.547219 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:47.547182    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" event={"ID":"b2a7f32e-9f27-4859-bba4-3c2659fd1c40","Type":"ContainerStarted","Data":"1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40"}
Jul 07 14:29:47.547604 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:47.547250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" event={"ID":"b2a7f32e-9f27-4859-bba4-3c2659fd1c40","Type":"ContainerStarted","Data":"18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1"}
Jul 07 14:29:47.547604 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:47.547437    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:47.573632 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:47.573582    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" podStartSLOduration=2.012846887 podStartE2EDuration="31.573565569s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.560718682s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:29:47.571690816 +0000 UTC m=+1233.680002409" watchObservedRunningTime="2026-07-07 14:29:47.573565569 +0000 UTC m=+1233.681877163"
Jul 07 14:29:48.552122 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:48.552077    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:29:54.562483 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:29:54.562446    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:31:16.440648 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:16.440616    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:31:26.381786 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:26.381744    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"]
Jul 07 14:31:26.382238 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:26.382022    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="kserve-container" containerID="cri-o://18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1" gracePeriod=30
Jul 07 14:31:26.382238 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:26.382089    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="kube-rbac-proxy" containerID="cri-o://1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40" gracePeriod=30
Jul 07 14:31:26.436525 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:26.436477    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 07 14:31:26.947511 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:26.947476    2569 generic.go:356] "Generic (PLEG): container finished" podID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerID="1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40" exitCode=2
Jul 07 14:31:26.947691 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:26.947548    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" event={"ID":"b2a7f32e-9f27-4859-bba4-3c2659fd1c40","Type":"ContainerDied","Data":"1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40"}
Jul 07 14:31:27.331621 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.331598    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:31:27.471183 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.471088    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kserve-provision-location\") pod \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") "
Jul 07 14:31:27.471183 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.471158    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-rbac-proxy-sar-config\") pod \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") "
Jul 07 14:31:27.471183 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.471187    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-proxy-tls\") pod \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") "
Jul 07 14:31:27.471749 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.471216    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxbq7\" (UniqueName: \"kubernetes.io/projected/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-api-access-qxbq7\") pod \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") "
Jul 07 14:31:27.471749 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.471249    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-cabundle-cert\") pod \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\" (UID: \"b2a7f32e-9f27-4859-bba4-3c2659fd1c40\") "
Jul 07 14:31:27.471749 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.471614    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2a7f32e-9f27-4859-bba4-3c2659fd1c40" (UID: "b2a7f32e-9f27-4859-bba4-3c2659fd1c40"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:31:27.471749 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.471666    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2a7f32e-9f27-4859-bba4-3c2659fd1c40" (UID: "b2a7f32e-9f27-4859-bba4-3c2659fd1c40"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:31:27.473396 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.473366    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-api-access-qxbq7" (OuterVolumeSpecName: "kube-api-access-qxbq7") pod "b2a7f32e-9f27-4859-bba4-3c2659fd1c40" (UID: "b2a7f32e-9f27-4859-bba4-3c2659fd1c40"). InnerVolumeSpecName "kube-api-access-qxbq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:31:27.473510 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.473415    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2a7f32e-9f27-4859-bba4-3c2659fd1c40" (UID: "b2a7f32e-9f27-4859-bba4-3c2659fd1c40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:31:27.475443 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.475417    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2a7f32e-9f27-4859-bba4-3c2659fd1c40" (UID: "b2a7f32e-9f27-4859-bba4-3c2659fd1c40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:31:27.572167 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.572131    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kserve-provision-location\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:31:27.572167 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.572161    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:31:27.572167 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.572171    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-proxy-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:31:27.572384 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.572180    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxbq7\" (UniqueName: \"kubernetes.io/projected/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-kube-api-access-qxbq7\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:31:27.572384 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.572189    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a7f32e-9f27-4859-bba4-3c2659fd1c40-cabundle-cert\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:31:27.954139 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.954089    2569 generic.go:356] "Generic (PLEG): container finished" podID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerID="18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1" exitCode=0
Jul 07 14:31:27.954340 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.954165    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"
Jul 07 14:31:27.954340 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.954178    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" event={"ID":"b2a7f32e-9f27-4859-bba4-3c2659fd1c40","Type":"ContainerDied","Data":"18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1"}
Jul 07 14:31:27.954340 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.954257    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6" event={"ID":"b2a7f32e-9f27-4859-bba4-3c2659fd1c40","Type":"ContainerDied","Data":"209c7110796527bd5f015d18f031075cc0c3a6e5730a9519fb3c097f836f8ff7"}
Jul 07 14:31:27.954340 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.954296    2569 scope.go:117] "RemoveContainer" containerID="1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40"
Jul 07 14:31:27.966387 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.966362    2569 scope.go:117] "RemoveContainer" containerID="18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1"
Jul 07 14:31:27.973938 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.973919    2569 scope.go:117] "RemoveContainer" containerID="cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0"
Jul 07 14:31:27.981459 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.981425    2569 scope.go:117] "RemoveContainer" containerID="1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40"
Jul 07 14:31:27.982203 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:31:27.982172    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40\": container with ID starting with 1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40 not found: ID does not exist" containerID="1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40"
Jul 07 14:31:27.982313 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.982215    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40"} err="failed to get container status \"1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40\": rpc error: code = NotFound desc = could not find container \"1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40\": container with ID starting with 1803faff0cc893fd0a96e922892067e1da13fb4f8a16a44474600ebb45badb40 not found: ID does not exist"
Jul 07 14:31:27.982313 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.982250    2569 scope.go:117] "RemoveContainer" containerID="18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1"
Jul 07 14:31:27.982593 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:31:27.982574    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1\": container with ID starting with 18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1 not found: ID does not exist" containerID="18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1"
Jul 07 14:31:27.982648 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.982600    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1"} err="failed to get container status \"18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1\": rpc error: code = NotFound desc = could not find container \"18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1\": container with ID starting with 18e5345768347b66c4a91922901c2798e88f924cfb80f97a9a6bac853f8ac1d1 not found: ID does not exist"
Jul 07 14:31:27.982648 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.982619    2569 scope.go:117] "RemoveContainer" containerID="cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0"
Jul 07 14:31:27.982884 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:31:27.982868    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0\": container with ID starting with cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0 not found: ID does not exist" containerID="cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0"
Jul 07 14:31:27.982934 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.982889    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0"} err="failed to get container status \"cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0\": rpc error: code = NotFound desc = could not find container \"cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0\": container with ID starting with cbe9eff1657e1ff0b269911fde0d490e9f794cc807f2c2403f7bb51dcc2af7f0 not found: ID does not exist"
Jul 07 14:31:27.983281 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.983263    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"]
Jul 07 14:31:27.985746 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:27.985722    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-mm4w6"]
Jul 07 14:31:28.440491 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:28.440457    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" path="/var/lib/kubelet/pods/b2a7f32e-9f27-4859-bba4-3c2659fd1c40/volumes"
Jul 07 14:31:31.755432 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.755400    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"]
Jul 07 14:31:31.755809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.755685    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="storage-initializer"
Jul 07 14:31:31.755809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.755696    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="storage-initializer"
Jul 07 14:31:31.755809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.755706    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="kube-rbac-proxy"
Jul 07 14:31:31.755809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.755712    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="kube-rbac-proxy"
Jul 07 14:31:31.755809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.755721    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="kserve-container"
Jul 07 14:31:31.755809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.755726    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="kserve-container"
Jul 07 14:31:31.755809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.755771    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="kserve-container"
Jul 07 14:31:31.755809 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.755779    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a7f32e-9f27-4859-bba4-3c2659fd1c40" containerName="kube-rbac-proxy"
Jul 07 14:31:31.760549 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.760532    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:31.763419 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.763396    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 07 14:31:31.763557 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.763404    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:31:31.764419 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.764377    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 07 14:31:31.764419 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.764391    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 14:31:31.764419 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.764409    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 14:31:31.764636 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.764451    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:31:31.764636 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.764399    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rswgx\""
Jul 07 14:31:31.778711 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.778684    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"]
Jul 07 14:31:31.906978 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.906940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:31.906978 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.906987    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mcf\" (UniqueName: \"kubernetes.io/projected/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-api-access-d9mcf\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:31.907244 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.907018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:31.907244 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.907101    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:31.907244 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:31.907225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.008529 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.008431    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.008529 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.008475    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.008752 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.008593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mcf\" (UniqueName: \"kubernetes.io/projected/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-api-access-d9mcf\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.008752 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.008626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.008752 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.008663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.008898 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.008878    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.009420 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.009395    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.009504 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.009395    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.011000 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.010981    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.021754 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.021732    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mcf\" (UniqueName: \"kubernetes.io/projected/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-api-access-d9mcf\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.070811 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.070775    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:32.228664 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.228628    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"]
Jul 07 14:31:32.236424 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:31:32.236396    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5996ae57_6222_4c5e_bdbb_a063d1e0d60a.slice/crio-d3d038ec118c65aa05b6abda0817927ec2dc7593a14ee084c13b78a2dfb6eca5 WatchSource:0}: Error finding container d3d038ec118c65aa05b6abda0817927ec2dc7593a14ee084c13b78a2dfb6eca5: Status 404 returned error can't find the container with id d3d038ec118c65aa05b6abda0817927ec2dc7593a14ee084c13b78a2dfb6eca5
Jul 07 14:31:32.975345 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.975305    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" event={"ID":"5996ae57-6222-4c5e-bdbb-a063d1e0d60a","Type":"ContainerStarted","Data":"95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d"}
Jul 07 14:31:32.975810 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:32.975368    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" event={"ID":"5996ae57-6222-4c5e-bdbb-a063d1e0d60a","Type":"ContainerStarted","Data":"d3d038ec118c65aa05b6abda0817927ec2dc7593a14ee084c13b78a2dfb6eca5"}
Jul 07 14:31:37.996127 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:37.996074    2569 generic.go:356] "Generic (PLEG): container finished" podID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerID="95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d" exitCode=0
Jul 07 14:31:37.996513 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:37.996152    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" event={"ID":"5996ae57-6222-4c5e-bdbb-a063d1e0d60a","Type":"ContainerDied","Data":"95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d"}
Jul 07 14:31:39.002967 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:39.002923    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" event={"ID":"5996ae57-6222-4c5e-bdbb-a063d1e0d60a","Type":"ContainerStarted","Data":"4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6"}
Jul 07 14:31:39.003392 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:39.003014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" event={"ID":"5996ae57-6222-4c5e-bdbb-a063d1e0d60a","Type":"ContainerStarted","Data":"c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8"}
Jul 07 14:31:39.003392 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:39.003282    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:39.028687 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:39.028628    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" podStartSLOduration=3.013707027 podStartE2EDuration="8.028606938s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.014899911s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:31:39.02825915 +0000 UTC m=+1345.136570743" watchObservedRunningTime="2026-07-07 14:31:39.028606938 +0000 UTC m=+1345.136918533"
Jul 07 14:31:40.007752 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:40.007715    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:31:46.017105 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:31:46.017071    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:33:10.441469 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:10.441389    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:33:11.776761 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:11.776717    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"]
Jul 07 14:33:12.392784 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:12.392710    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="kserve-container" containerID="cri-o://c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8" gracePeriod=30
Jul 07 14:33:12.392980 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:12.392727    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="kube-rbac-proxy" containerID="cri-o://4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6" gracePeriod=30
Jul 07 14:33:13.399501 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.399465    2569 generic.go:356] "Generic (PLEG): container finished" podID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerID="4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6" exitCode=2
Jul 07 14:33:13.399900 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.399539    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" event={"ID":"5996ae57-6222-4c5e-bdbb-a063d1e0d60a","Type":"ContainerDied","Data":"4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6"}
Jul 07 14:33:13.560047 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.559966    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:33:13.673028 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.672981    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-rbac-proxy-sar-config\") pod \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") "
Jul 07 14:33:13.673257 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.673057    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9mcf\" (UniqueName: \"kubernetes.io/projected/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-api-access-d9mcf\") pod \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") "
Jul 07 14:33:13.673257 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.673104    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kserve-provision-location\") pod \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") "
Jul 07 14:33:13.673257 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.673163    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-cabundle-cert\") pod \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") "
Jul 07 14:33:13.673257 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.673237    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-proxy-tls\") pod \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\" (UID: \"5996ae57-6222-4c5e-bdbb-a063d1e0d60a\") "
Jul 07 14:33:13.673514 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.673492    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5996ae57-6222-4c5e-bdbb-a063d1e0d60a" (UID: "5996ae57-6222-4c5e-bdbb-a063d1e0d60a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:33:13.673578 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.673530    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5996ae57-6222-4c5e-bdbb-a063d1e0d60a" (UID: "5996ae57-6222-4c5e-bdbb-a063d1e0d60a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:33:13.675377 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.675341    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5996ae57-6222-4c5e-bdbb-a063d1e0d60a" (UID: "5996ae57-6222-4c5e-bdbb-a063d1e0d60a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:33:13.675377 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.675360    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-api-access-d9mcf" (OuterVolumeSpecName: "kube-api-access-d9mcf") pod "5996ae57-6222-4c5e-bdbb-a063d1e0d60a" (UID: "5996ae57-6222-4c5e-bdbb-a063d1e0d60a"). InnerVolumeSpecName "kube-api-access-d9mcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:33:13.679699 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.679655    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5996ae57-6222-4c5e-bdbb-a063d1e0d60a" (UID: "5996ae57-6222-4c5e-bdbb-a063d1e0d60a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:33:13.774055 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.773996    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-proxy-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:33:13.774055 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.774046    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:33:13.774055 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.774061    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mcf\" (UniqueName: \"kubernetes.io/projected/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kube-api-access-d9mcf\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:33:13.774055 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.774070    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-kserve-provision-location\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:33:13.774055 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:13.774079    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5996ae57-6222-4c5e-bdbb-a063d1e0d60a-cabundle-cert\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:33:14.405987 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.405951    2569 generic.go:356] "Generic (PLEG): container finished" podID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerID="c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8" exitCode=0
Jul 07 14:33:14.406435 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.406003    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" event={"ID":"5996ae57-6222-4c5e-bdbb-a063d1e0d60a","Type":"ContainerDied","Data":"c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8"}
Jul 07 14:33:14.406435 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.406016    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"
Jul 07 14:33:14.406435 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.406058    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm" event={"ID":"5996ae57-6222-4c5e-bdbb-a063d1e0d60a","Type":"ContainerDied","Data":"d3d038ec118c65aa05b6abda0817927ec2dc7593a14ee084c13b78a2dfb6eca5"}
Jul 07 14:33:14.406435 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.406086    2569 scope.go:117] "RemoveContainer" containerID="4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6"
Jul 07 14:33:14.415269 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.415141    2569 scope.go:117] "RemoveContainer" containerID="c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8"
Jul 07 14:33:14.423786 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.423761    2569 scope.go:117] "RemoveContainer" containerID="95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d"
Jul 07 14:33:14.432424 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.432401    2569 scope.go:117] "RemoveContainer" containerID="4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6"
Jul 07 14:33:14.432752 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:33:14.432733    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6\": container with ID starting with 4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6 not found: ID does not exist" containerID="4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6"
Jul 07 14:33:14.432792 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.432762    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6"} err="failed to get container status \"4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6\": rpc error: code = NotFound desc = could not find container \"4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6\": container with ID starting with 4aa692cb71bc6865ebe814a4e20a116ea71293bb595214b987b712031c77fbc6 not found: ID does not exist"
Jul 07 14:33:14.432792 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.432780    2569 scope.go:117] "RemoveContainer" containerID="c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8"
Jul 07 14:33:14.433074 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:33:14.433053    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8\": container with ID starting with c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8 not found: ID does not exist" containerID="c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8"
Jul 07 14:33:14.433153 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.433080    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8"} err="failed to get container status \"c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8\": rpc error: code = NotFound desc = could not find container \"c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8\": container with ID starting with c4a296aa1ea32dd3b40c0e481c875d865bca0abbc53b26ce2f593db575cee1e8 not found: ID does not exist"
Jul 07 14:33:14.433153 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.433098    2569 scope.go:117] "RemoveContainer" containerID="95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d"
Jul 07 14:33:14.433378 ip-10-0-135-29 kubenswrapper[2569]: E0707 14:33:14.433355    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d\": container with ID starting with 95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d not found: ID does not exist" containerID="95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d"
Jul 07 14:33:14.433439 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.433390    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d"} err="failed to get container status \"95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d\": rpc error: code = NotFound desc = could not find container \"95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d\": container with ID starting with 95ca5d26a913ce4a79ef38e599eb270ab9b997b581297f98897c799e64b1eb3d not found: ID does not exist"
Jul 07 14:33:14.441320 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.441288    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"]
Jul 07 14:33:14.448856 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:14.448823    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-zb8mm"]
Jul 07 14:33:16.440072 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:16.440038    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" path="/var/lib/kubelet/pods/5996ae57-6222-4c5e-bdbb-a063d1e0d60a/volumes"
Jul 07 14:33:17.061000 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.060964    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"]
Jul 07 14:33:17.061309 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.061296    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="kube-rbac-proxy"
Jul 07 14:33:17.061375 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.061312    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="kube-rbac-proxy"
Jul 07 14:33:17.061375 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.061321    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="kserve-container"
Jul 07 14:33:17.061375 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.061327    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="kserve-container"
Jul 07 14:33:17.061375 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.061337    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="storage-initializer"
Jul 07 14:33:17.061375 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.061343    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="storage-initializer"
Jul 07 14:33:17.061517 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.061392    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="kserve-container"
Jul 07 14:33:17.061517 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.061405    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5996ae57-6222-4c5e-bdbb-a063d1e0d60a" containerName="kube-rbac-proxy"
Jul 07 14:33:17.068206 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.068170    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.077465 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.077442    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:33:17.078338 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.078317    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 07 14:33:17.078418 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.078362    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 07 14:33:17.078712 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.078688    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 14:33:17.078819 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.078733    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:33:17.079204 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.079189    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rswgx\""
Jul 07 14:33:17.079256 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.079221    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 14:33:17.100340 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.100303    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"]
Jul 07 14:33:17.202473 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.202429    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.202662 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.202532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.202662 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.202560    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0cec7cf-687b-459b-8d9c-bf26569a1a24-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.202662 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.202587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5fl\" (UniqueName: \"kubernetes.io/projected/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-api-access-gb5fl\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.202662 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.202609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.304009 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.303962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.304199 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.304034    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.304199 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.304059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0cec7cf-687b-459b-8d9c-bf26569a1a24-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.304199 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.304081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5fl\" (UniqueName: \"kubernetes.io/projected/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-api-access-gb5fl\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.304199 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.304099    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.304569 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.304544    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.304779 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.304750    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.304896 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.304794    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.306677 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.306652    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0cec7cf-687b-459b-8d9c-bf26569a1a24-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.321076 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.320994    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5fl\" (UniqueName: \"kubernetes.io/projected/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-api-access-gb5fl\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.379546 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.379499    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:17.529888 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:17.529851    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"]
Jul 07 14:33:18.424283 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:18.424242    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" event={"ID":"b0cec7cf-687b-459b-8d9c-bf26569a1a24","Type":"ContainerStarted","Data":"5a07882ac5119e77b101273267e4c739d11095ae3d0f61b2aab5d7f50fc5221a"}
Jul 07 14:33:18.424483 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:18.424314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" event={"ID":"b0cec7cf-687b-459b-8d9c-bf26569a1a24","Type":"ContainerStarted","Data":"f2848d91ac41c95ae2536a1834ce17f133ef7ba35a8c831da2534fccbdf821bc"}
Jul 07 14:33:48.541842 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:48.541806    2569 generic.go:356] "Generic (PLEG): container finished" podID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerID="5a07882ac5119e77b101273267e4c739d11095ae3d0f61b2aab5d7f50fc5221a" exitCode=0
Jul 07 14:33:48.542297 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:48.541879    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" event={"ID":"b0cec7cf-687b-459b-8d9c-bf26569a1a24","Type":"ContainerDied","Data":"5a07882ac5119e77b101273267e4c739d11095ae3d0f61b2aab5d7f50fc5221a"}
Jul 07 14:33:49.548727 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:49.548691    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" event={"ID":"b0cec7cf-687b-459b-8d9c-bf26569a1a24","Type":"ContainerStarted","Data":"71a579d6f20f1112fd448ec48d63e3f3c8b99f8eb7fc4851606e4b89c0817706"}
Jul 07 14:33:49.549250 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:49.548759    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" event={"ID":"b0cec7cf-687b-459b-8d9c-bf26569a1a24","Type":"ContainerStarted","Data":"461768fdbd169b2fbb1ac89d75dfc20293be9e980208176f2b6a82f6ce115b03"}
Jul 07 14:33:49.549250 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:49.548953    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:49.578598 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:49.578537    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" podStartSLOduration=2.300633704 podStartE2EDuration="32.57851947s" totalImagesPullingTime="0s" totalInitContainerRuntime="30.277885766s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:49.575966801 +0000 UTC m=+1475.684278394" watchObservedRunningTime="2026-07-07 14:33:49.57851947 +0000 UTC m=+1475.686831063"
Jul 07 14:33:50.552649 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:50.552615    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:33:56.564438 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:33:56.564404    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:34:14.433795 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:34:14.433763    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:34:14.434970 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:34:14.434943    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:35:22.440315 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:22.440281    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:35:27.089204 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:27.089167    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"]
Jul 07 14:35:27.089680 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:27.089405    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="kserve-container" containerID="cri-o://461768fdbd169b2fbb1ac89d75dfc20293be9e980208176f2b6a82f6ce115b03" gracePeriod=30
Jul 07 14:35:27.089680 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:27.089431    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="kube-rbac-proxy" containerID="cri-o://71a579d6f20f1112fd448ec48d63e3f3c8b99f8eb7fc4851606e4b89c0817706" gracePeriod=30
Jul 07 14:35:27.955241 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:27.955206    2569 generic.go:356] "Generic (PLEG): container finished" podID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerID="71a579d6f20f1112fd448ec48d63e3f3c8b99f8eb7fc4851606e4b89c0817706" exitCode=2
Jul 07 14:35:27.955241 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:27.955235    2569 generic.go:356] "Generic (PLEG): container finished" podID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerID="461768fdbd169b2fbb1ac89d75dfc20293be9e980208176f2b6a82f6ce115b03" exitCode=0
Jul 07 14:35:27.955442 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:27.955278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" event={"ID":"b0cec7cf-687b-459b-8d9c-bf26569a1a24","Type":"ContainerDied","Data":"71a579d6f20f1112fd448ec48d63e3f3c8b99f8eb7fc4851606e4b89c0817706"}
Jul 07 14:35:27.955442 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:27.955340    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" event={"ID":"b0cec7cf-687b-459b-8d9c-bf26569a1a24","Type":"ContainerDied","Data":"461768fdbd169b2fbb1ac89d75dfc20293be9e980208176f2b6a82f6ce115b03"}
Jul 07 14:35:28.041943 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.041918    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:35:28.122077 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.121991    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb5fl\" (UniqueName: \"kubernetes.io/projected/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-api-access-gb5fl\") pod \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") "
Jul 07 14:35:28.122077 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.122041    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-rbac-proxy-sar-config\") pod \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") "
Jul 07 14:35:28.122599 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.122142    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-cabundle-cert\") pod \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") "
Jul 07 14:35:28.122599 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.122185    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0cec7cf-687b-459b-8d9c-bf26569a1a24-proxy-tls\") pod \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") "
Jul 07 14:35:28.122599 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.122209    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kserve-provision-location\") pod \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\" (UID: \"b0cec7cf-687b-459b-8d9c-bf26569a1a24\") "
Jul 07 14:35:28.122599 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.122504    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0cec7cf-687b-459b-8d9c-bf26569a1a24" (UID: "b0cec7cf-687b-459b-8d9c-bf26569a1a24"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:35:28.122599 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.122542    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0cec7cf-687b-459b-8d9c-bf26569a1a24" (UID: "b0cec7cf-687b-459b-8d9c-bf26569a1a24"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:35:28.124319 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.124297    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-api-access-gb5fl" (OuterVolumeSpecName: "kube-api-access-gb5fl") pod "b0cec7cf-687b-459b-8d9c-bf26569a1a24" (UID: "b0cec7cf-687b-459b-8d9c-bf26569a1a24"). InnerVolumeSpecName "kube-api-access-gb5fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:35:28.124424 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.124397    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cec7cf-687b-459b-8d9c-bf26569a1a24-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0cec7cf-687b-459b-8d9c-bf26569a1a24" (UID: "b0cec7cf-687b-459b-8d9c-bf26569a1a24"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:35:28.127181 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.127160    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0cec7cf-687b-459b-8d9c-bf26569a1a24" (UID: "b0cec7cf-687b-459b-8d9c-bf26569a1a24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:35:28.223489 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.223434    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-cabundle-cert\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:35:28.223489 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.223480    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0cec7cf-687b-459b-8d9c-bf26569a1a24-proxy-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:35:28.223489 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.223491    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kserve-provision-location\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:35:28.223489 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.223500    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5fl\" (UniqueName: \"kubernetes.io/projected/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-api-access-gb5fl\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:35:28.223489 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.223508    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0cec7cf-687b-459b-8d9c-bf26569a1a24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:35:28.962015 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.961983    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"
Jul 07 14:35:28.962015 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.962005    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc" event={"ID":"b0cec7cf-687b-459b-8d9c-bf26569a1a24","Type":"ContainerDied","Data":"f2848d91ac41c95ae2536a1834ce17f133ef7ba35a8c831da2534fccbdf821bc"}
Jul 07 14:35:28.962273 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.962081    2569 scope.go:117] "RemoveContainer" containerID="71a579d6f20f1112fd448ec48d63e3f3c8b99f8eb7fc4851606e4b89c0817706"
Jul 07 14:35:28.970303 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.970284    2569 scope.go:117] "RemoveContainer" containerID="461768fdbd169b2fbb1ac89d75dfc20293be9e980208176f2b6a82f6ce115b03"
Jul 07 14:35:28.977517 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.977497    2569 scope.go:117] "RemoveContainer" containerID="5a07882ac5119e77b101273267e4c739d11095ae3d0f61b2aab5d7f50fc5221a"
Jul 07 14:35:28.985739 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.985718    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"]
Jul 07 14:35:28.990587 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:28.990567    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-qtcnc"]
Jul 07 14:35:30.441255 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:30.441223    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" path="/var/lib/kubelet/pods/b0cec7cf-687b-459b-8d9c-bf26569a1a24/volumes"
Jul 07 14:35:32.546163 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.546105    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"]
Jul 07 14:35:32.546547 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.546436    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="kserve-container"
Jul 07 14:35:32.546547 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.546448    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="kserve-container"
Jul 07 14:35:32.546547 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.546465    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="kube-rbac-proxy"
Jul 07 14:35:32.546547 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.546471    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="kube-rbac-proxy"
Jul 07 14:35:32.546547 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.546480    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="storage-initializer"
Jul 07 14:35:32.546547 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.546485    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="storage-initializer"
Jul 07 14:35:32.546547 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.546535    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="kserve-container"
Jul 07 14:35:32.546547 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.546543    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0cec7cf-687b-459b-8d9c-bf26569a1a24" containerName="kube-rbac-proxy"
Jul 07 14:35:32.549876 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.549856    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.552673 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.552639    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:35:32.553130 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.553099    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 07 14:35:32.553524 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.553498    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rswgx\""
Jul 07 14:35:32.553632 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.553541    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 07 14:35:32.553632 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.553568    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 14:35:32.553632 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.553592    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 14:35:32.553788 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.553652    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:35:32.577479 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.577444    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"]
Jul 07 14:35:32.656303 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.656258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.656494 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.656322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.656494 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.656407    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb68ae55-1110-490b-83a7-f60ecf41517c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.656494 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.656443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb68ae55-1110-490b-83a7-f60ecf41517c-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.656619 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.656498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbzj\" (UniqueName: \"kubernetes.io/projected/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-api-access-tvbzj\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.756921 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.756824    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb68ae55-1110-490b-83a7-f60ecf41517c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.756921 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.756868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb68ae55-1110-490b-83a7-f60ecf41517c-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.756921 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.756901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbzj\" (UniqueName: \"kubernetes.io/projected/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-api-access-tvbzj\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.756921 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.756924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.757309 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.756958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.757396 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.757367    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb68ae55-1110-490b-83a7-f60ecf41517c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.757719 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.757693    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.757798 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.757721    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.759373 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.759355    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb68ae55-1110-490b-83a7-f60ecf41517c-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.768683 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.768655    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbzj\" (UniqueName: \"kubernetes.io/projected/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-api-access-tvbzj\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:32.860024 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:32.859979    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:33.002303 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:33.002277    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"]
Jul 07 14:35:33.003797 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:35:33.003766    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb68ae55_1110_490b_83a7_f60ecf41517c.slice/crio-1f8f59553ac911430c56f8444e97377b3cda3b5755cfe0e678f0ad3a6e8a8aff WatchSource:0}: Error finding container 1f8f59553ac911430c56f8444e97377b3cda3b5755cfe0e678f0ad3a6e8a8aff: Status 404 returned error can't find the container with id 1f8f59553ac911430c56f8444e97377b3cda3b5755cfe0e678f0ad3a6e8a8aff
Jul 07 14:35:33.984616 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:33.984578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" event={"ID":"bb68ae55-1110-490b-83a7-f60ecf41517c","Type":"ContainerStarted","Data":"529964d929cfb76187d7244378c68ff47e84234ffbe04d40e243c55cb5ef83f5"}
Jul 07 14:35:33.985022 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:33.984637    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" event={"ID":"bb68ae55-1110-490b-83a7-f60ecf41517c","Type":"ContainerStarted","Data":"1f8f59553ac911430c56f8444e97377b3cda3b5755cfe0e678f0ad3a6e8a8aff"}
Jul 07 14:35:43.021338 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:43.021300    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerID="529964d929cfb76187d7244378c68ff47e84234ffbe04d40e243c55cb5ef83f5" exitCode=0
Jul 07 14:35:43.021739 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:43.021346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" event={"ID":"bb68ae55-1110-490b-83a7-f60ecf41517c","Type":"ContainerDied","Data":"529964d929cfb76187d7244378c68ff47e84234ffbe04d40e243c55cb5ef83f5"}
Jul 07 14:35:44.027503 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:44.027467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" event={"ID":"bb68ae55-1110-490b-83a7-f60ecf41517c","Type":"ContainerStarted","Data":"686557b5933e7fc8b4f06d9aa502c7bd3b7078dcecdb587f2fe164b6fc829bbf"}
Jul 07 14:35:44.027874 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:44.027533    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" event={"ID":"bb68ae55-1110-490b-83a7-f60ecf41517c","Type":"ContainerStarted","Data":"9d84569000e6de1c55b1762aa78adb5c3d281136d1da47a0f537208dde0553d7"}
Jul 07 14:35:44.027874 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:44.027741    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:44.055282 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:44.055231    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" podStartSLOduration=2.912438977 podStartE2EDuration="12.055211665s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.142772688s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:44.053068313 +0000 UTC m=+1590.161379908" watchObservedRunningTime="2026-07-07 14:35:44.055211665 +0000 UTC m=+1590.163523259"
Jul 07 14:35:45.031476 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:45.031443    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:45.032771 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:45.032746    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 07 14:35:46.035024 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:46.034967    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 07 14:35:51.039880 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:51.039847    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:35:51.040495 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:35:51.040467    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 07 14:36:01.041327 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:01.041287    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:36:02.387211 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:02.387176    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"]
Jul 07 14:36:02.387633 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:02.387385    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kserve-container" containerID="cri-o://9d84569000e6de1c55b1762aa78adb5c3d281136d1da47a0f537208dde0553d7" gracePeriod=30
Jul 07 14:36:02.387633 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:02.387438    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kube-rbac-proxy" containerID="cri-o://686557b5933e7fc8b4f06d9aa502c7bd3b7078dcecdb587f2fe164b6fc829bbf" gracePeriod=30
Jul 07 14:36:03.105224 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.105196    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerID="686557b5933e7fc8b4f06d9aa502c7bd3b7078dcecdb587f2fe164b6fc829bbf" exitCode=2
Jul 07 14:36:03.105224 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.105224    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerID="9d84569000e6de1c55b1762aa78adb5c3d281136d1da47a0f537208dde0553d7" exitCode=0
Jul 07 14:36:03.105364 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.105259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" event={"ID":"bb68ae55-1110-490b-83a7-f60ecf41517c","Type":"ContainerDied","Data":"686557b5933e7fc8b4f06d9aa502c7bd3b7078dcecdb587f2fe164b6fc829bbf"}
Jul 07 14:36:03.105364 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.105317    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" event={"ID":"bb68ae55-1110-490b-83a7-f60ecf41517c","Type":"ContainerDied","Data":"9d84569000e6de1c55b1762aa78adb5c3d281136d1da47a0f537208dde0553d7"}
Jul 07 14:36:03.234576 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.234549    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:36:03.322255 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.322163    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb68ae55-1110-490b-83a7-f60ecf41517c-kserve-provision-location\") pod \"bb68ae55-1110-490b-83a7-f60ecf41517c\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") "
Jul 07 14:36:03.322255 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.322208    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb68ae55-1110-490b-83a7-f60ecf41517c-proxy-tls\") pod \"bb68ae55-1110-490b-83a7-f60ecf41517c\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") "
Jul 07 14:36:03.322502 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.322263    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-cabundle-cert\") pod \"bb68ae55-1110-490b-83a7-f60ecf41517c\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") "
Jul 07 14:36:03.322502 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.322307    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvbzj\" (UniqueName: \"kubernetes.io/projected/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-api-access-tvbzj\") pod \"bb68ae55-1110-490b-83a7-f60ecf41517c\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") "
Jul 07 14:36:03.322502 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.322422    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-rbac-proxy-sar-config\") pod \"bb68ae55-1110-490b-83a7-f60ecf41517c\" (UID: \"bb68ae55-1110-490b-83a7-f60ecf41517c\") "
Jul 07 14:36:03.322721 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.322692    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bb68ae55-1110-490b-83a7-f60ecf41517c" (UID: "bb68ae55-1110-490b-83a7-f60ecf41517c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:36:03.322781 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.322755    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bb68ae55-1110-490b-83a7-f60ecf41517c" (UID: "bb68ae55-1110-490b-83a7-f60ecf41517c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:36:03.323225 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.323203    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb68ae55-1110-490b-83a7-f60ecf41517c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb68ae55-1110-490b-83a7-f60ecf41517c" (UID: "bb68ae55-1110-490b-83a7-f60ecf41517c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 14:36:03.324460 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.324437    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb68ae55-1110-490b-83a7-f60ecf41517c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb68ae55-1110-490b-83a7-f60ecf41517c" (UID: "bb68ae55-1110-490b-83a7-f60ecf41517c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:36:03.324557 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.324472    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-api-access-tvbzj" (OuterVolumeSpecName: "kube-api-access-tvbzj") pod "bb68ae55-1110-490b-83a7-f60ecf41517c" (UID: "bb68ae55-1110-490b-83a7-f60ecf41517c"). InnerVolumeSpecName "kube-api-access-tvbzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:36:03.423817 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.423785    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-cabundle-cert\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:36:03.423817 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.423815    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvbzj\" (UniqueName: \"kubernetes.io/projected/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-api-access-tvbzj\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:36:03.423817 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.423824    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb68ae55-1110-490b-83a7-f60ecf41517c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:36:03.424278 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.423833    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb68ae55-1110-490b-83a7-f60ecf41517c-kserve-provision-location\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:36:03.424278 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:03.423843    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb68ae55-1110-490b-83a7-f60ecf41517c-proxy-tls\") on node \"ip-10-0-135-29.ec2.internal\" DevicePath \"\""
Jul 07 14:36:04.111171 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:04.111144    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"
Jul 07 14:36:04.111345 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:04.111144    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4" event={"ID":"bb68ae55-1110-490b-83a7-f60ecf41517c","Type":"ContainerDied","Data":"1f8f59553ac911430c56f8444e97377b3cda3b5755cfe0e678f0ad3a6e8a8aff"}
Jul 07 14:36:04.111345 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:04.111297    2569 scope.go:117] "RemoveContainer" containerID="686557b5933e7fc8b4f06d9aa502c7bd3b7078dcecdb587f2fe164b6fc829bbf"
Jul 07 14:36:04.120006 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:04.119986    2569 scope.go:117] "RemoveContainer" containerID="9d84569000e6de1c55b1762aa78adb5c3d281136d1da47a0f537208dde0553d7"
Jul 07 14:36:04.127095 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:04.127075    2569 scope.go:117] "RemoveContainer" containerID="529964d929cfb76187d7244378c68ff47e84234ffbe04d40e243c55cb5ef83f5"
Jul 07 14:36:04.140509 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:04.140490    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"]
Jul 07 14:36:04.147125 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:04.147093    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-ts6g4"]
Jul 07 14:36:04.440429 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:04.440400    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" path="/var/lib/kubelet/pods/bb68ae55-1110-490b-83a7-f60ecf41517c/volumes"
Jul 07 14:36:42.017759 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:42.017728    2569 ???:1] "http: TLS handshake error from 10.0.130.166:49358: EOF"
Jul 07 14:36:42.023194 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:42.023174    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zhwzv_b5f534d4-5cbf-4bbe-af34-41916ed91208/global-pull-secret-syncer/0.log"
Jul 07 14:36:42.276412 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:42.276338    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lffsb_65a0ecad-5ba4-47c7-879d-3fd9aae552fa/konnectivity-agent/0.log"
Jul 07 14:36:42.340402 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:42.340378    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-29.ec2.internal_479338554c92fbd8bd392d8bacd28b2d/haproxy/0.log"
Jul 07 14:36:46.962882 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:46.962849    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-ldmlk_f52588df-a9cd-4b8a-b936-9bd1eff9002f/cluster-monitoring-operator/0.log"
Jul 07 14:36:47.568285 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:47.568259    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r7qg8_52656e2f-b141-4ea4-88a0-4ec26954c9a3/node-exporter/0.log"
Jul 07 14:36:47.647299 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:47.647276    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r7qg8_52656e2f-b141-4ea4-88a0-4ec26954c9a3/kube-rbac-proxy/0.log"
Jul 07 14:36:47.715759 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:47.715736    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r7qg8_52656e2f-b141-4ea4-88a0-4ec26954c9a3/init-textfile/0.log"
Jul 07 14:36:50.262448 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.262415    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"]
Jul 07 14:36:50.262812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.262694    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="storage-initializer"
Jul 07 14:36:50.262812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.262705    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="storage-initializer"
Jul 07 14:36:50.262812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.262712    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kube-rbac-proxy"
Jul 07 14:36:50.262812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.262719    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kube-rbac-proxy"
Jul 07 14:36:50.262812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.262728    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kserve-container"
Jul 07 14:36:50.262812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.262733    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kserve-container"
Jul 07 14:36:50.262812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.262804    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kube-rbac-proxy"
Jul 07 14:36:50.262812 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.262816    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb68ae55-1110-490b-83a7-f60ecf41517c" containerName="kserve-container"
Jul 07 14:36:50.265899 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.265876    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.270378 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.270356    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z7x97\"/\"kube-root-ca.crt\""
Jul 07 14:36:50.270991 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.270972    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z7x97\"/\"openshift-service-ca.crt\""
Jul 07 14:36:50.271082 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.271011    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z7x97\"/\"default-dockercfg-4m829\""
Jul 07 14:36:50.286622 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.286597    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"]
Jul 07 14:36:50.385802 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.385769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxsr\" (UniqueName: \"kubernetes.io/projected/a4929d7a-482d-4ad0-8576-7bb5a852db69-kube-api-access-jmxsr\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.385932 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.385810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-sys\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.385932 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.385827    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-proc\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.385932 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.385843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-podres\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.385932 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.385861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-lib-modules\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.487126 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.487072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxsr\" (UniqueName: \"kubernetes.io/projected/a4929d7a-482d-4ad0-8576-7bb5a852db69-kube-api-access-jmxsr\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.487261 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.487143    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-sys\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.487261 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.487168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-proc\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.487261 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.487197    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-podres\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.487261 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.487225    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-lib-modules\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.487438 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.487266    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-sys\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.487438 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.487286    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-proc\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.487438 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.487338    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-podres\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.487438 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.487373    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4929d7a-482d-4ad0-8576-7bb5a852db69-lib-modules\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.496958 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.496937    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxsr\" (UniqueName: \"kubernetes.io/projected/a4929d7a-482d-4ad0-8576-7bb5a852db69-kube-api-access-jmxsr\") pod \"perf-node-gather-daemonset-xctnb\" (UID: \"a4929d7a-482d-4ad0-8576-7bb5a852db69\") " pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.575671 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.575608    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:50.709481 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:50.709449    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"]
Jul 07 14:36:50.712346 ip-10-0-135-29 kubenswrapper[2569]: W0707 14:36:50.712318    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda4929d7a_482d_4ad0_8576_7bb5a852db69.slice/crio-ea0b9246d52d42e542c23b6125124e9c5cec80530df04c434f27c9cd0ee80163 WatchSource:0}: Error finding container ea0b9246d52d42e542c23b6125124e9c5cec80530df04c434f27c9cd0ee80163: Status 404 returned error can't find the container with id ea0b9246d52d42e542c23b6125124e9c5cec80530df04c434f27c9cd0ee80163
Jul 07 14:36:51.046521 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:51.046491    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/2.log"
Jul 07 14:36:51.050479 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:51.050459    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-rcshd_806d472c-532b-49fa-a1d6-c2a047ae8c3b/console-operator/3.log"
Jul 07 14:36:51.293488 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:51.293451    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb" event={"ID":"a4929d7a-482d-4ad0-8576-7bb5a852db69","Type":"ContainerStarted","Data":"a023387db1ccdb48c279a37226d96d47dc1a851ef8e79a497f0f8a3b553365af"}
Jul 07 14:36:51.293841 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:51.293529    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb" event={"ID":"a4929d7a-482d-4ad0-8576-7bb5a852db69","Type":"ContainerStarted","Data":"ea0b9246d52d42e542c23b6125124e9c5cec80530df04c434f27c9cd0ee80163"}
Jul 07 14:36:51.293841 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:51.293581    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:51.316346 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:51.316252    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb" podStartSLOduration=1.316235898 podStartE2EDuration="1.316235898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:36:51.31565947 +0000 UTC m=+1657.423971067" watchObservedRunningTime="2026-07-07 14:36:51.316235898 +0000 UTC m=+1657.424547773"
Jul 07 14:36:52.231476 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:52.231450    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-fff64_8b8ff1b0-b894-4838-818f-829a14e7d677/volume-data-source-validator/0.log"
Jul 07 14:36:53.301207 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:53.301173    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2kvq8_27acbc3b-ba1e-411b-a893-a78af377f3ad/dns/0.log"
Jul 07 14:36:53.343352 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:53.343323    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2kvq8_27acbc3b-ba1e-411b-a893-a78af377f3ad/kube-rbac-proxy/0.log"
Jul 07 14:36:53.581925 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:53.581845    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cfxrl_4963cb5e-680a-4bbf-8e68-258d0f5b7331/dns-node-resolver/0.log"
Jul 07 14:36:54.356636 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:54.356604    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5pz8_70208e8b-d198-4a27-abb8-248b9a6b9234/node-ca/0.log"
Jul 07 14:36:55.430381 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:55.430355    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-596dffc677-vvdpt_75cb0ebd-abf1-489a-97e9-88324f9cf349/router/0.log"
Jul 07 14:36:55.869831 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:55.869762    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9snb9_41251dd9-eb69-46b2-a2c3-fbabac0dc673/serve-healthcheck-canary/0.log"
Jul 07 14:36:56.477329 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:56.477302    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-k8t8j_85998ef5-484f-4de6-9dc5-7ded0e7317cd/insights-operator/0.log"
Jul 07 14:36:56.477688 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:56.477589    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-k8t8j_85998ef5-484f-4de6-9dc5-7ded0e7317cd/insights-operator/1.log"
Jul 07 14:36:56.544341 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:56.544312    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-45gjt_6aadd26c-2770-473f-8987-aed718ce85d1/kube-rbac-proxy/0.log"
Jul 07 14:36:56.611594 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:56.611572    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-45gjt_6aadd26c-2770-473f-8987-aed718ce85d1/exporter/0.log"
Jul 07 14:36:56.660622 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:56.660600    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-45gjt_6aadd26c-2770-473f-8987-aed718ce85d1/extractor/0.log"
Jul 07 14:36:57.306990 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:57.306962    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z7x97/perf-node-gather-daemonset-xctnb"
Jul 07 14:36:59.284547 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:59.284519    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-lkq96_11a9ed90-2d3f-4ccd-a4ae-ec5824677d33/manager/0.log"
Jul 07 14:36:59.322753 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:59.322727    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-ckpm2_6dc1e5f4-d961-4fd7-9145-773705c9f344/server/0.log"
Jul 07 14:36:59.715047 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:36:59.715019    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-6wtqp_067647ef-0ba5-4d3b-a9fb-800a34f49e00/seaweedfs/0.log"
Jul 07 14:37:05.695732 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:05.695693    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-p4txj_ba2eb9cf-295b-4d9f-aee2-28ffa025e24c/kube-storage-version-migrator-operator/1.log"
Jul 07 14:37:05.696592 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:05.696570    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-p4txj_ba2eb9cf-295b-4d9f-aee2-28ffa025e24c/kube-storage-version-migrator-operator/0.log"
Jul 07 14:37:06.800726 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:06.800696    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-785hx_e0933ede-8054-466c-82a5-83ad1751743b/kube-multus/0.log"
Jul 07 14:37:06.841583 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:06.841560    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4mkfq_6cab4996-d1b9-4425-ba8a-c5889b50acfb/kube-multus-additional-cni-plugins/0.log"
Jul 07 14:37:06.887242 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:06.887217    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4mkfq_6cab4996-d1b9-4425-ba8a-c5889b50acfb/egress-router-binary-copy/0.log"
Jul 07 14:37:06.918527 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:06.918507    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4mkfq_6cab4996-d1b9-4425-ba8a-c5889b50acfb/cni-plugins/0.log"
Jul 07 14:37:06.954415 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:06.954382    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4mkfq_6cab4996-d1b9-4425-ba8a-c5889b50acfb/bond-cni-plugin/0.log"
Jul 07 14:37:06.984005 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:06.983989    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4mkfq_6cab4996-d1b9-4425-ba8a-c5889b50acfb/routeoverride-cni/0.log"
Jul 07 14:37:07.015710 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:07.015694    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4mkfq_6cab4996-d1b9-4425-ba8a-c5889b50acfb/whereabouts-cni-bincopy/0.log"
Jul 07 14:37:07.052967 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:07.052910    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4mkfq_6cab4996-d1b9-4425-ba8a-c5889b50acfb/whereabouts-cni/0.log"
Jul 07 14:37:07.683510 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:07.683458    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8jm6t_4298379e-f6c7-44eb-b922-f393e007c4af/network-metrics-daemon/0.log"
Jul 07 14:37:07.726005 ip-10-0-135-29 kubenswrapper[2569]: I0707 14:37:07.725977    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8jm6t_4298379e-f6c7-44eb-b922-f393e007c4af/kube-rbac-proxy/0.log"