Jul 28 12:53:19.289497 ip-10-0-142-83 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:53:19.667451 ip-10-0-142-83 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:19.667451 ip-10-0-142-83 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:53:19.667451 ip-10-0-142-83 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:19.667451 ip-10-0-142-83 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:53:19.667451 ip-10-0-142-83 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:19.668170 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.667517    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:53:19.672156 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672139    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:19.672156 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672154    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:19.672156 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672158    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672161    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672166    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672169    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672172    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672174    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672178    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672180    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672183    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672186    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672188    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672191    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672199    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672202    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672204    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672207    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672209    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672212    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672236    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672239    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:19.672272 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672242    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672244    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672247    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672250    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672252    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672255    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672258    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672261    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672263    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672266    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672268    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672271    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672274    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672277    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672280    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672283    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672285    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672288    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672291    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672293    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:19.672747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672296    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672299    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672301    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672304    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672307    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672309    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672312    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672314    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672317    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672319    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672321    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672324    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672326    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672329    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672332    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672335    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672338    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672340    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672343    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672345    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:19.673258 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672348    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672350    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672353    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672356    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672359    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672361    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672364    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672368    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672372    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672375    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672378    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672381    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672384    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672389    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672394    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672397    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672400    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672403    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672405    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:19.673747 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672408    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672411    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672414    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672417    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672420    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672422    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672425    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672428    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672431    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672434    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672438    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672442    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672445    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672448    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672451    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672454    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672457    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.672460    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673166    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:19.674225 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673173    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673176    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673179    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673182    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673185    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673187    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673190    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673193    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673196    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673199    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673202    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673204    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673207    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673209    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673212    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673226    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673229    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673232    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673235    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:19.674685 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673237    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673241    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673244    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673246    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673251    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673255    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673258    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673262    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673265    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673268    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673271    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673274    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673277    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673280    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673282    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673285    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673288    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673290    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673300    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:19.675237 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673303    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673306    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673309    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673312    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673315    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673317    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673320    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673323    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673325    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673328    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673330    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673333    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673336    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673338    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673341    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673344    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673346    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673349    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673351    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673354    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:19.675705 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673356    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673359    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673361    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673364    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673366    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673369    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673372    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673375    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673378    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673380    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673383    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673385    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673389    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673393    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673396    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673399    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673402    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673405    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673407    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:19.676181 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673410    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673412    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673415    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673417    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673421    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673423    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673426    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673429    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673431    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673434    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673436    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673439    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673441    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673444    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673446    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673452    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673455    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673458    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673462    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673465    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:19.676678 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673468    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673857    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673862    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673866    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673869    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673872    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673876    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673879    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673882    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673885    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673887    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673890    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673893    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673895    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673898    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673901    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673903    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673906    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673908    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673911    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:19.677166 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673914    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673917    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673919    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673922    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673924    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673927    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673929    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673933    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673937    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673939    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673942    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673946    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673949    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673951    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673954    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673956    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673959    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673962    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673965    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:19.677666 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673968    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673970    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673973    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673975    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673978    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673981    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673984    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673986    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673989    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673991    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673994    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673996    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.673999    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674001    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674004    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674007    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674009    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674013    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674017    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:19.678179 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674021    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674024    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674027    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674030    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674032    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674035    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674038    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674040    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674043    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674045    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674048    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674053    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674056    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674058    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674061    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674064    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674067    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674069    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674072    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674076    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:19.678660 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674079    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674081    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674084    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674086    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674089    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674091    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674094    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674096    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674099    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674101    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674104    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674106    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674109    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674112    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674114    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674116    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674119    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674122    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674124    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:19.679145 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674126    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674129    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674131    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674206    2572 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674230    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674239    2572 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674245    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674252    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674257    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674263    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674268    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674271    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674275    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674279    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674282    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674285    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674288    2572 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674291    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674294    2572 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674297    2572 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674300    2572 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674304    2572 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674307    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674310    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:53:19.679648 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674313    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674317    2572 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674320    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674323    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674326    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674329    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674332    2572 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674335    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674338    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674341    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674344    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674348    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674352    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674355    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674358    2572 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674361    2572 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674365    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674369    2572 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674372    2572 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674375    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674380    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674383    2572 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674387    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674389    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674392    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:53:19.680237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674395    2572 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674398    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674401    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674404    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674407    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674410    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674412    2572 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674415    2572 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674419    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674422    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674424    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674427    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674431    2572 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674434    2572 flags.go:64] FLAG: --help="false"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674437    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-142-83.ec2.internal"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674440    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674443    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674446    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674449    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674454    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674457    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674459    2572 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674462    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:53:19.680846 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674465    2572 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674468    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674472    2572 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674474    2572 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674479    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674482    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674485    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674488    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674490    2572 flags.go:64] FLAG: --lock-file=""
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674494    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674497    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674500    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674505    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674507    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674510    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674513    2572 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674516    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674519    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674522    2572 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674524    2572 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674529    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674532    2572 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674536    2572 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674539    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674541    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:53:19.681479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674544    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674547    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674550    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674554    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674557    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674564    2572 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674567    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674570    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674573    2572 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674576    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674581    2572 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674585    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674589    2572 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674591    2572 flags.go:64] FLAG: --port="10250"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674594    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674597    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05cf13cd1a888efa4"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674601    2572 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674603    2572 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674606    2572 flags.go:64] FLAG: --register-node="true"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674609    2572 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674613    2572 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674616    2572 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674618    2572 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674621    2572 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674625    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:53:19.682113 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674628    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674631    2572 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674634    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674637    2572 flags.go:64] FLAG: --runonce="false"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674640    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674643    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674646    2572 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674648    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674651    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674654    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674657    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674662    2572 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674665    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674668    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674670    2572 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674673    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674676    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674679    2572 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674684    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674690    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674693    2572 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674696    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674700    2572 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674703    2572 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674705    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:53:19.682749 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674708    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674711    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674714    2572 flags.go:64] FLAG: --v="2"
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674718    2572 flags.go:64] FLAG: --version="false"
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674725    2572 flags.go:64] FLAG: --vmodule=""
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674729    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.674732    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674846    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674849    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674852    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674855    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674858    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674862    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674865    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674867    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674870    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674872    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674875    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674879    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674881    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674884    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:19.683412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674887    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674889    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674892    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674894    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674900    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674903    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674905    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674908    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674910    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674913    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674915    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674918    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674921    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674923    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674927    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674931    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674933    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674936    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674939    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:19.683972 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674941    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674944    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674946    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674949    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674951    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674955    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674959    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674962    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674965    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674968    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674973    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674976    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674978    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674981    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674984    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674987    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674990    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674993    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674996    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.674999    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:19.684443 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675002    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675005    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675007    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675009    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675012    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675015    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675017    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675020    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675023    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675026    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675028    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675031    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675034    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675036    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675039    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675041    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675044    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675047    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675050    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675052    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:19.684923 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675055    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675057    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675061    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675064    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675066    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675069    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675072    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675074    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675077    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675080    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675082    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675087    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675090    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675093    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675096    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675099    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675101    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675104    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675106    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:19.685427 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675109    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675111    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675114    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675117    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675119    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675121    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.675124    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.675781    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.682655    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.682672    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.682690    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.682696    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682752    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682757    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682761    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:19.685902 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682764    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682767    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682770    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682773    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682776    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682779    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682781    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682784    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682786    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682789    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682791    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682794    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682798    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682801    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682803    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682806    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682808    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682811    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682813    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:19.686425 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682816    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682820    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682824    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682827    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682830    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682833    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682835    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682839    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682842    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682845    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682847    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682850    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682852    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682855    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682858    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682861    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682863    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682866    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682868    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682871    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:19.686881 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682874    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682876    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682879    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682881    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682884    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682887    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682889    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682892    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682894    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682897    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682900    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682902    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682905    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682907    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682910    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682913    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682915    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682918    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682921    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:19.687375 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682924    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682927    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682930    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682932    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682936    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682940    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682942    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682945    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682948    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682950    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682953    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682955    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682958    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682960    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682963    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682965    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682967    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682970    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682973    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682975    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:19.687908 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682978    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682980    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682983    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682985    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682988    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682990    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682993    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682995    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.682998    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683000    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683003    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683006    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683010    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683014    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683017    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683020    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683022    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:19.688412 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683025    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.683030    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683151    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683158    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683162    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683165    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683168    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683171    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683174    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683176    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683179    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683181    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683184    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683186    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683189    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:19.688898 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683192    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683195    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683197    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683200    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683202    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683206    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683210    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683213    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683227    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683229    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683232    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683235    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683238    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683240    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683243    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683246    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683249    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683252    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683254    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:19.689283 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683257    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683259    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683262    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683265    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683267    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683269    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683272    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683275    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683277    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683280    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683282    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683285    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683288    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683290    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683293    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683296    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683298    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683300    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683303    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683306    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:19.689778 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683308    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683311    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683313    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683316    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683319    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683322    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683324    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683327    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683330    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683334    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683338    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683341    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683344    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683347    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683350    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683352    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683355    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683358    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683360    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683363    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:19.691029 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683366    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683368    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683371    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683373    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683375    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683378    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683381    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683383    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683386    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683388    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683391    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683393    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683395    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683399    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683401    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683404    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683406    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683409    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683411    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683414    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:19.691527 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683418    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683421    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683429    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683432    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683434    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683437    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:19.683440    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.683445    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.683556    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.685865    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.687572    2572 server.go:1019] "Starting client certificate rotation"
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.687692    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.688578    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:53:19.692002 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.690554    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:53:19.695770 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.695745    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:53:19.710331 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.710310    2572 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:53:19.712694 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.712680    2572 log.go:25] "Validated CRI v1 image API"
Jul 28 12:53:19.713509 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.713494    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:53:19.719466 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.719438    2572 fs.go:135] Filesystem UUIDs: map[635dd876-da00-48b1-a675-2d52007fa2b4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d72f8396-fc49-477e-89ce-5e9c23d33e12:/dev/nvme0n1p4]
Jul 28 12:53:19.719532 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.719466    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:53:19.722909 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.722891    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:53:19.724836 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.724722    2572 manager.go:216] Machine: {Timestamp:2026-07-28 12:53:19.723364395 +0000 UTC m=+0.321080292 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098707 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23df8882d524b9101865c5f7102cc5 SystemUUID:ec23df88-82d5-24b9-1018-65c5f7102cc5 BootID:8cc7f693-d483-4202-a9ce-b22ad3e63201 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:08:9e:48:b6:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:08:9e:48:b6:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:7f:e7:6b:ee:22 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:53:19.724836 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.724833    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:53:19.724949 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.724937    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:53:19.728477 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.728445    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:53:19.728626 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.728482    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-83.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:53:19.730000 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.729989    2572 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:53:19.730039 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.730002    2572 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:53:19.730039 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.730015    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:53:19.730818 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.730807    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:53:19.732064 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.732054    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:53:19.734910 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.734898    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:53:19.735036 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.735026    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:53:19.735076 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.735058    2572 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:53:19.735076 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.735069    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:53:19.735140 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.735086    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:53:19.735140 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.735095    2572 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:53:19.735140 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.735110    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:53:19.735505 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.735488    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:53:19.735505 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.735507    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:53:19.739043 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.739023    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:53:19.741563 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.741547    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:53:19.741638 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.741578    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:53:19.743611 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743598    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:53:19.743666 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743618    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:53:19.743666 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743627    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:53:19.743666 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743635    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:53:19.743666 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743644    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:53:19.743666 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743651    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:53:19.743666 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743660    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:53:19.743666 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743665    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:53:19.743843 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743672    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:53:19.743843 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743677    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:53:19.743843 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743685    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:53:19.743843 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.743690    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:53:19.744574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.744565    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:53:19.744574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.744574    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:53:19.745506 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.745482    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:53:19.745558 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.745525    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:53:19.747627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.747614    2572 server.go:1268] "Started kubelet"
Jul 28 12:53:19.747883 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.747730    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:53:19.747883 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.747837    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:53:19.748002 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.747781    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:53:19.752863 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.752835    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:53:19.752985 ip-10-0-142-83 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:53:19.755299 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.755282    2572 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:53:19.758636 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.758612    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:53:19.760205 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.760188    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-83.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:53:19.761230 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.760158    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-83.ec2.internal.18c6753b46a2fe7c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-83.ec2.internal,UID:ip-10-0-142-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-83.ec2.internal,},FirstTimestamp:2026-07-28 12:53:19.747587708 +0000 UTC m=+0.345303602,LastTimestamp:2026-07-28 12:53:19.747587708 +0000 UTC m=+0.345303602,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-83.ec2.internal,}"
Jul 28 12:53:19.766066 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.766052    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:53:19.766066 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.766059    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:53:19.766168 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.766111    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:19.766832 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.766810    2572 factory.go:55] Registering systemd factory
Jul 28 12:53:19.766832 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.766833    2572 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:53:19.766965 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.766916    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:53:19.766965 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.766932    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:53:19.767037 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.766919    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:53:19.767096 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.767084    2572 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:53:19.767127 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.767097    2572 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:53:19.767244 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.767212    2572 factory.go:153] Registering CRI-O factory
Jul 28 12:53:19.767298 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.767249    2572 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:53:19.767347 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.767334    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:53:19.767393 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.767360    2572 factory.go:103] Registering Raw factory
Jul 28 12:53:19.767393 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.767374    2572 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:53:19.767662 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.767643    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 12:53:19.767746 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.767725    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 12:53:19.767796 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.767783    2572 manager.go:349] Starting recovery of all containers
Jul 28 12:53:19.772196 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.772173    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ch6nd"
Jul 28 12:53:19.777689 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.777674    2572 manager.go:354] Recovery completed
Jul 28 12:53:19.783140 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.783129    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:19.783388 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.783365    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ch6nd"
Jul 28 12:53:19.785246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.785230    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:19.785382 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.785264    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:19.785382 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.785281    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:19.786052 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.786035    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:53:19.786052 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.786049    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:53:19.786193 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.786054    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:19.786193 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.786064    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:53:19.787277 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.787262    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:19.787368 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.787300    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:19.787368 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.787316    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:19.789198 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.789182    2572 policy_none.go:49] "None policy: Start"
Jul 28 12:53:19.789267 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.789205    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:53:19.789267 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.789234    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:53:19.791233 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.791208    2572 policy_none.go:47] "Start"
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.830353    2572 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.830390    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.830404    2572 server.go:87] "Starting device plugin registration server"
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.830675    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.830686    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.830783    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.830879    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.830887    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.831453    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:53:19.844847 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.831510    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:19.898416 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.898378    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:53:19.899532 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.899506    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:53:19.899631 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.899541    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:53:19.899631 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.899566    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:53:19.899631 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.899599    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:53:19.902553 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.902533    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:19.931398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.931294    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:19.932340 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.932319    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:19.932460 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.932351    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:19.932460 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.932363    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:19.932460 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.932386    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-83.ec2.internal"
Jul 28 12:53:19.941118 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:19.941095    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-83.ec2.internal"
Jul 28 12:53:19.941118 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.941117    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-83.ec2.internal\": node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:19.966365 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:19.966335    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:20.000516 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.000479    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"]
Jul 28 12:53:20.000680 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.000566    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:20.002647 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.002629    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:20.002741 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.002657    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:20.002741 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.002671    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:20.005091 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.005078    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:20.005196 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.005183    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.005257 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.005231    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:20.006415 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.006398    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:20.006495 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.006428    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:20.006495 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.006439    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:20.006495 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.006458    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:20.006495 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.006480    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:20.006495 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.006494    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:20.008776 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.008761    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.008841 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.008793    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:20.009773 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.009754    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:20.009877 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.009785    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:20.009877 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.009804    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:20.029983 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.029958    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-83.ec2.internal\" not found" node="ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.034411 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.034396    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-83.ec2.internal\" not found" node="ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.066813 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.066786    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:20.068994 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.068977    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e36d7aafc98cad33ea701597790b088-config\") pod \"kube-apiserver-proxy-ip-10-0-142-83.ec2.internal\" (UID: \"5e36d7aafc98cad33ea701597790b088\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.069057 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.069001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/755ebb668ab847a4c2017edcdd11706f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal\" (UID: \"755ebb668ab847a4c2017edcdd11706f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.069057 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.069026    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/755ebb668ab847a4c2017edcdd11706f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal\" (UID: \"755ebb668ab847a4c2017edcdd11706f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.167832 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.167789    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:20.169963 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.169948    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e36d7aafc98cad33ea701597790b088-config\") pod \"kube-apiserver-proxy-ip-10-0-142-83.ec2.internal\" (UID: \"5e36d7aafc98cad33ea701597790b088\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.170017 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.169971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/755ebb668ab847a4c2017edcdd11706f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal\" (UID: \"755ebb668ab847a4c2017edcdd11706f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.170017 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.169987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/755ebb668ab847a4c2017edcdd11706f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal\" (UID: \"755ebb668ab847a4c2017edcdd11706f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.170107 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.170025    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/755ebb668ab847a4c2017edcdd11706f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal\" (UID: \"755ebb668ab847a4c2017edcdd11706f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.170107 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.170067    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/755ebb668ab847a4c2017edcdd11706f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal\" (UID: \"755ebb668ab847a4c2017edcdd11706f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.170107 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.170064    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e36d7aafc98cad33ea701597790b088-config\") pod \"kube-apiserver-proxy-ip-10-0-142-83.ec2.internal\" (UID: \"5e36d7aafc98cad33ea701597790b088\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.268690 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.268598    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:20.332961 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.332939    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.338138 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.337864    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.369187 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.369149    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:20.469900 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.469851    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:20.570039 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.569957    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-83.ec2.internal\" not found"
Jul 28 12:53:20.570303 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.570285    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:20.666446 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.666407    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.677609 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.677586    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:53:20.679547 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.679533    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.687061 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.687045    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:53:20.687187 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.687169    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:53:20.687293 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.687266    2572 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a8270be37693043b592324c250e83a42-01f51b489b2b5514.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.142.83:52928->34.237.126.13:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal"
Jul 28 12:53:20.735602 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.735570    2572 apiserver.go:52] "Watching apiserver"
Jul 28 12:53:20.741849 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.741828    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:53:20.742203 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.742184    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-p6smg","kube-system/konnectivity-agent-q7rl6","kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4","openshift-dns/node-resolver-szhdp","openshift-image-registry/node-ca-ksgqt","openshift-multus/network-metrics-daemon-k4prs","openshift-network-operator/iptables-alerter-tvzl9","openshift-cluster-node-tuning-operator/tuned-stdct","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal","openshift-multus/multus-additional-cni-plugins-mpcpt","openshift-multus/multus-bwszv","openshift-network-diagnostics/network-check-target-lkksh","openshift-ovn-kubernetes/ovnkube-node-h4krx"]
Jul 28 12:53:20.744659 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.744641    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:20.744742 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.744706    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:20.746770 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.746753    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:20.749997 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.749972    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:53:20.749997 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.749969    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:53:20.750159 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.750015    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mm6mg\""
Jul 28 12:53:20.750546 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.750525    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:20.751993 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.751974    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:53:20.752284 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.752270    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:53:20.752358 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.752317    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dmsjd\""
Jul 28 12:53:20.752423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.752411    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:53:20.752705 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.752689    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:20.754426 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.754409    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:53:20.754496 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.754440    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rsznk\""
Jul 28 12:53:20.754538 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.754497    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:53:20.754832 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.754820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:20.756458 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.756444    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:53:20.756687 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.756673    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:53:20.756781 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.756768    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:53:20.756824 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.756772    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dqrr6\""
Jul 28 12:53:20.756985 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.756971    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:20.757039 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.757024    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:20.759118 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.759105    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:20.760808 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.760789    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4dnwb\""
Jul 28 12:53:20.760971 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.760955    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:53:20.761058 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.761013    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:53:20.761141 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.761059    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:53:20.761318 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.761300    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.762853 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.762833    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:53:20.763128 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.763115    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:53:20.763201 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.763164    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r4mt6\""
Jul 28 12:53:20.764092 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.763542    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:20.765621 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.765605    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:53:20.765621 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.765616    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:53:20.765759 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.765639    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:53:20.765759 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.765705    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5mf46\""
Jul 28 12:53:20.765913 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.765897    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.765972 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.765917    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:53:20.766025 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.766014    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:53:20.766142 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.766127    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:53:20.767528 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.767512    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-trbw7\""
Jul 28 12:53:20.768046 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.768031    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:53:20.768250 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.768237    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:20.768321 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:20.768289    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:20.770460 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.770442    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:20.772889 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772532    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:53:20.772889 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772546    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:53:20.772889 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772546    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:53:20.772889 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772635    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:53:20.772889 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-system-cni-dir\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.772889 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772792    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wt2rq\""
Jul 28 12:53:20.772889 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772794    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0e02d46-2f01-4d45-b048-f0dc22cfad61-host\") pod \"node-ca-ksgqt\" (UID: \"c0e02d46-2f01-4d45-b048-f0dc22cfad61\") " pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:20.772889 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk9ww\" (UniqueName: \"kubernetes.io/projected/c0e02d46-2f01-4d45-b048-f0dc22cfad61-kube-api-access-nk9ww\") pod \"node-ca-ksgqt\" (UID: \"c0e02d46-2f01-4d45-b048-f0dc22cfad61\") " pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:20.772889 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772894    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:53:20.773239 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772920    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-sysctl-conf\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.773239 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-socket-dir-parent\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.773239 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772961    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-run-netns\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.773239 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.772974    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-cnibin\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.773394 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773008    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64386eb9-8899-410e-adf0-3f21a2a226b0-dbus\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773565    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773572    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad991f88-1a5a-4745-b3c6-7f00b9f4ee96-tmp-dir\") pod \"node-resolver-szhdp\" (UID: \"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96\") " pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773649    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-modprobe-d\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773674    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8581eb58-b73d-4cc3-983c-585bb739933c-tmp\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-cni-binary-copy\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773719    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773765    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-sys\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773803    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-host\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-sys-fs\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773895    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-hostroot\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.773924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8581eb58-b73d-4cc3-983c-585bb739933c-etc-tuned\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw9fw\" (UniqueName: \"kubernetes.io/projected/8581eb58-b73d-4cc3-983c-585bb739933c-kube-api-access-xw9fw\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774044    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-run-k8s-cni-cncf-io\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774082    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-conf-dir\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774125    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11b94fe7-fd89-41af-b09b-79cfcb8afad8-host-slash\") pod \"iptables-alerter-tvzl9\" (UID: \"11b94fe7-fd89-41af-b09b-79cfcb8afad8\") " pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774200    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-sysctl-d\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.774627 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774249    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xkx\" (UniqueName: \"kubernetes.io/projected/bd90f16b-a738-49f8-99f8-ac9981959493-kube-api-access-h6xkx\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:20.775515 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774301    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-var-lib-cni-multus\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.775515 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774370    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-systemd\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.775515 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774412    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad991f88-1a5a-4745-b3c6-7f00b9f4ee96-hosts-file\") pod \"node-resolver-szhdp\" (UID: \"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96\") " pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:20.775515 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774488    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:20.775515 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774536    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnrt7\" (UniqueName: \"kubernetes.io/projected/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-kube-api-access-nnrt7\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:20.775515 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774560    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:20.775515 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.774590    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11b94fe7-fd89-41af-b09b-79cfcb8afad8-iptables-alerter-script\") pod \"iptables-alerter-tvzl9\" (UID: \"11b94fe7-fd89-41af-b09b-79cfcb8afad8\") " pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:20.776958 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.776939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-kubernetes\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.777035 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.776981    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-socket-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:20.777035 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777014    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-registration-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:20.777114 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777037    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-os-release\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.777114 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777065    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-daemon-config\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.777114 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777079    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:20.777244 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777130    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-run-multus-certs\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.777244 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777165    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89sp5\" (UniqueName: \"kubernetes.io/projected/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-kube-api-access-89sp5\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.777244 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777182    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zx9\" (UniqueName: \"kubernetes.io/projected/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-kube-api-access-f5zx9\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:20.777244 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777202    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-sysconfig\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.777385 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777293    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-cni-dir\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.777655 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-os-release\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:20.777710 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd90f16b-a738-49f8-99f8-ac9981959493-cni-binary-copy\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:20.777710 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777693    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:20.777811 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64386eb9-8899-410e-adf0-3f21a2a226b0-kubelet-config\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:20.777811 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777750    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-lib-modules\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.777811 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777780    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-system-cni-dir\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:20.777954 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777809    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-var-lib-cni-bin\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.777954 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777833    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b52c3e58-14e8-442c-b277-6ba3718e5682-konnectivity-ca\") pod \"konnectivity-agent-q7rl6\" (UID: \"b52c3e58-14e8-442c-b277-6ba3718e5682\") " pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:20.777954 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777857    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfvb\" (UniqueName: \"kubernetes.io/projected/ad991f88-1a5a-4745-b3c6-7f00b9f4ee96-kube-api-access-8mfvb\") pod \"node-resolver-szhdp\" (UID: \"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96\") " pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:20.777954 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777878    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-run\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.777954 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777902    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-device-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:20.777954 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777927    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd90f16b-a738-49f8-99f8-ac9981959493-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:20.777954 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777951    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-var-lib-kubelet\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.778275 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777974    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:20.778275 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.777998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd90f16b-a738-49f8-99f8-ac9981959493-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:20.778275 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.778021    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:20.778275 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.778044    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-etc-kubernetes\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:20.778275 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.778064    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-var-lib-kubelet\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:20.778275 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.778093    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b52c3e58-14e8-442c-b277-6ba3718e5682-agent-certs\") pod \"konnectivity-agent-q7rl6\" (UID: \"b52c3e58-14e8-442c-b277-6ba3718e5682\") " pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:20.778275 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.778117    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0e02d46-2f01-4d45-b048-f0dc22cfad61-serviceca\") pod \"node-ca-ksgqt\" (UID: \"c0e02d46-2f01-4d45-b048-f0dc22cfad61\") " pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:20.778275 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.778144    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r275l\" (UniqueName: \"kubernetes.io/projected/11b94fe7-fd89-41af-b09b-79cfcb8afad8-kube-api-access-r275l\") pod \"iptables-alerter-tvzl9\" (UID: \"11b94fe7-fd89-41af-b09b-79cfcb8afad8\") " pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:20.778275 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.778168    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-cnibin\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:20.782114 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.782096    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:53:20.794756 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.794703    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:48:21 +0000 UTC" deadline="2028-03-23 07:04:35.817389126 +0000 UTC"
Jul 28 12:53:20.794756 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.794752    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14490h11m15.022638923s"
Jul 28 12:53:20.803472 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.803449    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hwwdt"
Jul 28 12:53:20.813307 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:20.813287    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hwwdt"
Jul 28 12:53:22.142513 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.142492    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:53:22.153368 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153331    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:22.153368 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11b94fe7-fd89-41af-b09b-79cfcb8afad8-iptables-alerter-script\") pod \"iptables-alerter-tvzl9\" (UID: \"11b94fe7-fd89-41af-b09b-79cfcb8afad8\") " pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:22.153580 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.153477    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:22.153580 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153497    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-kubernetes\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.153580 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153527    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-socket-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.153580 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.153547    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs podName:43fa5143-75d6-4f79-88d0-97b5a5ecf09c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:22.653527174 +0000 UTC m=+1.976520092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs") pod "network-metrics-daemon-k4prs" (UID: "43fa5143-75d6-4f79-88d0-97b5a5ecf09c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:22.153580 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-registration-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153603    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-os-release\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153605    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-kubernetes\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153641    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-socket-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153644    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-daemon-config\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153664    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-os-release\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-run-multus-certs\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153695    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89sp5\" (UniqueName: \"kubernetes.io/projected/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-kube-api-access-89sp5\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153700    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-registration-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-run-ovn\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153758    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/66460849-0183-481b-a998-ac5bc9c81e35-ovnkube-config\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153760    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-run-multus-certs\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153860    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/66460849-0183-481b-a998-ac5bc9c81e35-ovn-node-metrics-cert\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.153894 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153892    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jms\" (UniqueName: \"kubernetes.io/projected/66460849-0183-481b-a998-ac5bc9c81e35-kube-api-access-c6jms\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153905    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11b94fe7-fd89-41af-b09b-79cfcb8afad8-iptables-alerter-script\") pod \"iptables-alerter-tvzl9\" (UID: \"11b94fe7-fd89-41af-b09b-79cfcb8afad8\") " pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153926    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zx9\" (UniqueName: \"kubernetes.io/projected/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-kube-api-access-f5zx9\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-sysconfig\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.153993    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-cni-dir\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154010    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-run-netns\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154025    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-etc-openvswitch\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154049    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-os-release\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd90f16b-a738-49f8-99f8-ac9981959493-cni-binary-copy\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154092    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-sysconfig\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154159    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-os-release\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154095    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154176    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-cni-dir\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154200    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64386eb9-8899-410e-adf0-3f21a2a226b0-kubelet-config\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-lib-modules\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154234    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154247    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-system-cni-dir\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.154574 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154273    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-var-lib-cni-bin\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154284    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/64386eb9-8899-410e-adf0-3f21a2a226b0-kubelet-config\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154294    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-daemon-config\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154307    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-lib-modules\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154298    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b52c3e58-14e8-442c-b277-6ba3718e5682-konnectivity-ca\") pod \"konnectivity-agent-q7rl6\" (UID: \"b52c3e58-14e8-442c-b277-6ba3718e5682\") " pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154340    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-var-lib-cni-bin\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154345    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-system-cni-dir\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154362    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfvb\" (UniqueName: \"kubernetes.io/projected/ad991f88-1a5a-4745-b3c6-7f00b9f4ee96-kube-api-access-8mfvb\") pod \"node-resolver-szhdp\" (UID: \"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96\") " pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-run\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154413    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-device-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154464    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-device-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154511    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-run\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154560    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd90f16b-a738-49f8-99f8-ac9981959493-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154625    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-var-lib-kubelet\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154640    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd90f16b-a738-49f8-99f8-ac9981959493-cni-binary-copy\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154655    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-kubelet\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-run-systemd\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.155423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154748    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-var-lib-kubelet\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154768    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154799    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd90f16b-a738-49f8-99f8-ac9981959493-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154836    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154851    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-log-socket\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154878    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-cni-netd\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154902    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-etc-kubernetes\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154966    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-var-lib-openvswitch\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.154995    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-etc-kubernetes\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155012    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-run-openvswitch\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155037    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/66460849-0183-481b-a998-ac5bc9c81e35-env-overrides\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155063    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/66460849-0183-481b-a998-ac5bc9c81e35-ovnkube-script-lib\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155088    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-var-lib-kubelet\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155098    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd90f16b-a738-49f8-99f8-ac9981959493-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155111    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-slash\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-cni-bin\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.156246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155158    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-var-lib-kubelet\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b52c3e58-14e8-442c-b277-6ba3718e5682-agent-certs\") pod \"konnectivity-agent-q7rl6\" (UID: \"b52c3e58-14e8-442c-b277-6ba3718e5682\") " pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155202    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0e02d46-2f01-4d45-b048-f0dc22cfad61-serviceca\") pod \"node-ca-ksgqt\" (UID: \"c0e02d46-2f01-4d45-b048-f0dc22cfad61\") " pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155260    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r275l\" (UniqueName: \"kubernetes.io/projected/11b94fe7-fd89-41af-b09b-79cfcb8afad8-kube-api-access-r275l\") pod \"iptables-alerter-tvzl9\" (UID: \"11b94fe7-fd89-41af-b09b-79cfcb8afad8\") " pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-cnibin\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155315    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-system-cni-dir\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155324    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd90f16b-a738-49f8-99f8-ac9981959493-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155338    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0e02d46-2f01-4d45-b048-f0dc22cfad61-host\") pod \"node-ca-ksgqt\" (UID: \"c0e02d46-2f01-4d45-b048-f0dc22cfad61\") " pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155362    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk9ww\" (UniqueName: \"kubernetes.io/projected/c0e02d46-2f01-4d45-b048-f0dc22cfad61-kube-api-access-nk9ww\") pod \"node-ca-ksgqt\" (UID: \"c0e02d46-2f01-4d45-b048-f0dc22cfad61\") " pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155388    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-sysctl-conf\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155388    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0e02d46-2f01-4d45-b048-f0dc22cfad61-host\") pod \"node-ca-ksgqt\" (UID: \"c0e02d46-2f01-4d45-b048-f0dc22cfad61\") " pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155360    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b52c3e58-14e8-442c-b277-6ba3718e5682-konnectivity-ca\") pod \"konnectivity-agent-q7rl6\" (UID: \"b52c3e58-14e8-442c-b277-6ba3718e5682\") " pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155451    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-socket-dir-parent\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155480    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-run-netns\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155490    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd90f16b-a738-49f8-99f8-ac9981959493-cnibin\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155506    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-run-ovn-kubernetes\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155527    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155542    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-system-cni-dir\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.157042 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155543    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-cnibin\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155564    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-socket-dir-parent\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155573    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0e02d46-2f01-4d45-b048-f0dc22cfad61-serviceca\") pod \"node-ca-ksgqt\" (UID: \"c0e02d46-2f01-4d45-b048-f0dc22cfad61\") " pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155582    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64386eb9-8899-410e-adf0-3f21a2a226b0-dbus\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155595    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-cnibin\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155605    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad991f88-1a5a-4745-b3c6-7f00b9f4ee96-tmp-dir\") pod \"node-resolver-szhdp\" (UID: \"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96\") " pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155627    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-modprobe-d\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8581eb58-b73d-4cc3-983c-585bb739933c-tmp\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155628    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-run-netns\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155663    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/64386eb9-8899-410e-adf0-3f21a2a226b0-dbus\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155685    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-sysctl-conf\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155709    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-systemd-units\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155728    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-cni-binary-copy\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155765    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155786    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-modprobe-d\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155824    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-sys\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155848    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-host\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.155869    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:22.157893 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155907    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-sys-fs\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.155919    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret podName:64386eb9-8899-410e-adf0-3f21a2a226b0 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:22.655902841 +0000 UTC m=+1.978895765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret") pod "global-pull-secret-syncer-p6smg" (UID: "64386eb9-8899-410e-adf0-3f21a2a226b0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155918    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad991f88-1a5a-4745-b3c6-7f00b9f4ee96-tmp-dir\") pod \"node-resolver-szhdp\" (UID: \"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96\") " pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155869    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-sys-fs\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155966    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-sys\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.155990    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-hostroot\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156007    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-host\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8581eb58-b73d-4cc3-983c-585bb739933c-etc-tuned\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156041    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw9fw\" (UniqueName: \"kubernetes.io/projected/8581eb58-b73d-4cc3-983c-585bb739933c-kube-api-access-xw9fw\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156051    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-hostroot\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-run-k8s-cni-cncf-io\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156089    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-conf-dir\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156123    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-run-k8s-cni-cncf-io\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156134    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-multus-conf-dir\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-node-log\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156173    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11b94fe7-fd89-41af-b09b-79cfcb8afad8-host-slash\") pod \"iptables-alerter-tvzl9\" (UID: \"11b94fe7-fd89-41af-b09b-79cfcb8afad8\") " pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156192    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-sysctl-d\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.158731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156217    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xkx\" (UniqueName: \"kubernetes.io/projected/bd90f16b-a738-49f8-99f8-ac9981959493-kube-api-access-h6xkx\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156227    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-cni-binary-copy\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156244    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-var-lib-cni-multus\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156269    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156291    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-host-var-lib-cni-multus\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156246    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11b94fe7-fd89-41af-b09b-79cfcb8afad8-host-slash\") pod \"iptables-alerter-tvzl9\" (UID: \"11b94fe7-fd89-41af-b09b-79cfcb8afad8\") " pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156314    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-systemd\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156337    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad991f88-1a5a-4745-b3c6-7f00b9f4ee96-hosts-file\") pod \"node-resolver-szhdp\" (UID: \"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96\") " pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156360    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156373    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-sysctl-d\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156387    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrt7\" (UniqueName: \"kubernetes.io/projected/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-kube-api-access-nnrt7\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156436    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156436    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad991f88-1a5a-4745-b3c6-7f00b9f4ee96-hosts-file\") pod \"node-resolver-szhdp\" (UID: \"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96\") " pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.156459    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8581eb58-b73d-4cc3-983c-585bb739933c-etc-systemd\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.158647    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8581eb58-b73d-4cc3-983c-585bb739933c-etc-tuned\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.158788    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8581eb58-b73d-4cc3-983c-585bb739933c-tmp\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.159314 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.158958    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b52c3e58-14e8-442c-b277-6ba3718e5682-agent-certs\") pod \"konnectivity-agent-q7rl6\" (UID: \"b52c3e58-14e8-442c-b277-6ba3718e5682\") " pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:22.160550 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.160536    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:22.160600 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.160553    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:22.160600 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.160562    2572 projected.go:196] Error preparing data for projected volume kube-api-access-rt77l for pod openshift-network-diagnostics/network-check-target-lkksh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:22.160660 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.160609    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l podName:64cdd13d-520d-4f12-86fa-a5f99d362fba nodeName:}" failed. No retries permitted until 2026-07-28 12:53:22.660598557 +0000 UTC m=+1.983591476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rt77l" (UniqueName: "kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l") pod "network-check-target-lkksh" (UID: "64cdd13d-520d-4f12-86fa-a5f99d362fba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:22.163388 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.163363    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfvb\" (UniqueName: \"kubernetes.io/projected/ad991f88-1a5a-4745-b3c6-7f00b9f4ee96-kube-api-access-8mfvb\") pod \"node-resolver-szhdp\" (UID: \"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96\") " pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:22.163565 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.163509    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r275l\" (UniqueName: \"kubernetes.io/projected/11b94fe7-fd89-41af-b09b-79cfcb8afad8-kube-api-access-r275l\") pod \"iptables-alerter-tvzl9\" (UID: \"11b94fe7-fd89-41af-b09b-79cfcb8afad8\") " pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:22.164031 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.164003    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk9ww\" (UniqueName: \"kubernetes.io/projected/c0e02d46-2f01-4d45-b048-f0dc22cfad61-kube-api-access-nk9ww\") pod \"node-ca-ksgqt\" (UID: \"c0e02d46-2f01-4d45-b048-f0dc22cfad61\") " pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:22.164124 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.164036    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw9fw\" (UniqueName: \"kubernetes.io/projected/8581eb58-b73d-4cc3-983c-585bb739933c-kube-api-access-xw9fw\") pod \"tuned-stdct\" (UID: \"8581eb58-b73d-4cc3-983c-585bb739933c\") " pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.164220 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.164193    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89sp5\" (UniqueName: \"kubernetes.io/projected/f6b221b1-965c-43c6-aacb-9ba92b12d4a9-kube-api-access-89sp5\") pod \"multus-bwszv\" (UID: \"f6b221b1-965c-43c6-aacb-9ba92b12d4a9\") " pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.164729 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.164708    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zx9\" (UniqueName: \"kubernetes.io/projected/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-kube-api-access-f5zx9\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:22.164912 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.164893    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrt7\" (UniqueName: \"kubernetes.io/projected/cbe8cb75-1a5f-4927-914a-e7e7c7c80350-kube-api-access-nnrt7\") pod \"aws-ebs-csi-driver-node-8nsd4\" (UID: \"cbe8cb75-1a5f-4927-914a-e7e7c7c80350\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.165760 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.165725    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xkx\" (UniqueName: \"kubernetes.io/projected/bd90f16b-a738-49f8-99f8-ac9981959493-kube-api-access-h6xkx\") pod \"multus-additional-cni-plugins-mpcpt\" (UID: \"bd90f16b-a738-49f8-99f8-ac9981959493\") " pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.177003 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.176977    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwszv"
Jul 28 12:53:22.257728 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257697    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-run-ovn\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.257728 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257729    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/66460849-0183-481b-a998-ac5bc9c81e35-ovnkube-config\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.257964 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257746    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/66460849-0183-481b-a998-ac5bc9c81e35-ovn-node-metrics-cert\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.257964 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257764    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jms\" (UniqueName: \"kubernetes.io/projected/66460849-0183-481b-a998-ac5bc9c81e35-kube-api-access-c6jms\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.257964 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257780    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-run-netns\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.257964 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257801    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-etc-openvswitch\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.257964 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257851    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-run-ovn\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.257964 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257863    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-kubelet\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.257964 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257884    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-run-systemd\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.257964 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257911    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-etc-openvswitch\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.257985    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-log-socket\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258010    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-cni-netd\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258032    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-var-lib-openvswitch\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258056    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-run-openvswitch\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258079    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/66460849-0183-481b-a998-ac5bc9c81e35-env-overrides\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258102    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/66460849-0183-481b-a998-ac5bc9c81e35-ovnkube-script-lib\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258129    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-slash\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258158    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-cni-bin\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258159    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-kubelet\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258193    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-run-ovn-kubernetes\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258210    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-cni-netd\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258225    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-systemd-units\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258264    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-run-netns\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-node-log\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258312    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-systemd-units\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258325    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.258356 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258349    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-slash\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258365    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/66460849-0183-481b-a998-ac5bc9c81e35-ovnkube-config\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258412    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-run-ovn-kubernetes\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258322    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-run-openvswitch\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258163    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-var-lib-openvswitch\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258226    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-log-socket\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258514    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-cni-bin\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258552    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-run-systemd\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258731    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-node-log\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258742    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66460849-0183-481b-a998-ac5bc9c81e35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.258832    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/66460849-0183-481b-a998-ac5bc9c81e35-env-overrides\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.259396 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.259121    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/66460849-0183-481b-a998-ac5bc9c81e35-ovnkube-script-lib\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.260914 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.260893    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/66460849-0183-481b-a998-ac5bc9c81e35-ovn-node-metrics-cert\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.269255 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.269225    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755ebb668ab847a4c2017edcdd11706f.slice/crio-2b5bc4ccb9cacb2de48efc515186c5653bb8222f7ddc833585fcb15350dd7018 WatchSource:0}: Error finding container 2b5bc4ccb9cacb2de48efc515186c5653bb8222f7ddc833585fcb15350dd7018: Status 404 returned error can't find the container with id 2b5bc4ccb9cacb2de48efc515186c5653bb8222f7ddc833585fcb15350dd7018
Jul 28 12:53:22.269814 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.269798    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e36d7aafc98cad33ea701597790b088.slice/crio-eb773264f8b392f0989f2dcbd7ecc23bc26eb01827d2ff15048064ecfcf8b2b2 WatchSource:0}: Error finding container eb773264f8b392f0989f2dcbd7ecc23bc26eb01827d2ff15048064ecfcf8b2b2: Status 404 returned error can't find the container with id eb773264f8b392f0989f2dcbd7ecc23bc26eb01827d2ff15048064ecfcf8b2b2
Jul 28 12:53:22.270471 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.270455    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jms\" (UniqueName: \"kubernetes.io/projected/66460849-0183-481b-a998-ac5bc9c81e35-kube-api-access-c6jms\") pod \"ovnkube-node-h4krx\" (UID: \"66460849-0183-481b-a998-ac5bc9c81e35\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.273867 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.273854    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:53:22.348719 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.348678    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:22.358488 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.358462    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52c3e58_14e8_442c_b277_6ba3718e5682.slice/crio-48daf7380bf0f8e1407376ba9019fa32d9b051817bf69087e13d87d1f217d5a1 WatchSource:0}: Error finding container 48daf7380bf0f8e1407376ba9019fa32d9b051817bf69087e13d87d1f217d5a1: Status 404 returned error can't find the container with id 48daf7380bf0f8e1407376ba9019fa32d9b051817bf69087e13d87d1f217d5a1
Jul 28 12:53:22.373879 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.373859    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4"
Jul 28 12:53:22.382522 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.382497    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe8cb75_1a5f_4927_914a_e7e7c7c80350.slice/crio-3f3b08893fb2f31fe17e9e8910b0660dab826b36d2bf469f40a3b8bb3d92769b WatchSource:0}: Error finding container 3f3b08893fb2f31fe17e9e8910b0660dab826b36d2bf469f40a3b8bb3d92769b: Status 404 returned error can't find the container with id 3f3b08893fb2f31fe17e9e8910b0660dab826b36d2bf469f40a3b8bb3d92769b
Jul 28 12:53:22.393190 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.393171    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-szhdp"
Jul 28 12:53:22.403149 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.403125    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad991f88_1a5a_4745_b3c6_7f00b9f4ee96.slice/crio-cc60d5e8f206ca43fd03d5dae687eccb4bd22d091380ac2448cd4eb14f9c3bf8 WatchSource:0}: Error finding container cc60d5e8f206ca43fd03d5dae687eccb4bd22d091380ac2448cd4eb14f9c3bf8: Status 404 returned error can't find the container with id cc60d5e8f206ca43fd03d5dae687eccb4bd22d091380ac2448cd4eb14f9c3bf8
Jul 28 12:53:22.408157 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.408140    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ksgqt"
Jul 28 12:53:22.416339 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.416317    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e02d46_2f01_4d45_b048_f0dc22cfad61.slice/crio-f7ebe1346e4075eb8dedd9a324cd96e90ed168ae323ede7083bf3551b64089df WatchSource:0}: Error finding container f7ebe1346e4075eb8dedd9a324cd96e90ed168ae323ede7083bf3551b64089df: Status 404 returned error can't find the container with id f7ebe1346e4075eb8dedd9a324cd96e90ed168ae323ede7083bf3551b64089df
Jul 28 12:53:22.423481 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.423460    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tvzl9"
Jul 28 12:53:22.431665 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.431640    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b94fe7_fd89_41af_b09b_79cfcb8afad8.slice/crio-dbe84bc6b10d3fe8b8fe6c8d314bae0f27d1ae0092a781620750e0406a2c2f5c WatchSource:0}: Error finding container dbe84bc6b10d3fe8b8fe6c8d314bae0f27d1ae0092a781620750e0406a2c2f5c: Status 404 returned error can't find the container with id dbe84bc6b10d3fe8b8fe6c8d314bae0f27d1ae0092a781620750e0406a2c2f5c
Jul 28 12:53:22.439160 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.439141    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-stdct"
Jul 28 12:53:22.446777 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.446754    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mpcpt"
Jul 28 12:53:22.447044 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.447024    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8581eb58_b73d_4cc3_983c_585bb739933c.slice/crio-a7224b21e9dcaef593d687289cc2241cad3efd23a084b81619970517f6c7c47d WatchSource:0}: Error finding container a7224b21e9dcaef593d687289cc2241cad3efd23a084b81619970517f6c7c47d: Status 404 returned error can't find the container with id a7224b21e9dcaef593d687289cc2241cad3efd23a084b81619970517f6c7c47d
Jul 28 12:53:22.454487 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.454463    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd90f16b_a738_49f8_99f8_ac9981959493.slice/crio-bb6f506956b1c361d6121366e54220125728ff746f0de410fbb6b185352f6e06 WatchSource:0}: Error finding container bb6f506956b1c361d6121366e54220125728ff746f0de410fbb6b185352f6e06: Status 404 returned error can't find the container with id bb6f506956b1c361d6121366e54220125728ff746f0de410fbb6b185352f6e06
Jul 28 12:53:22.486134 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.486108    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:22.495439 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.495404    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66460849_0183_481b_a998_ac5bc9c81e35.slice/crio-66e7bcccb5d2daed4f14e8c3a698884a81e47ab9658502166e90aad027a56c7f WatchSource:0}: Error finding container 66e7bcccb5d2daed4f14e8c3a698884a81e47ab9658502166e90aad027a56c7f: Status 404 returned error can't find the container with id 66e7bcccb5d2daed4f14e8c3a698884a81e47ab9658502166e90aad027a56c7f
Jul 28 12:53:22.545760 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.545729    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:22.628680 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:22.628649    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b221b1_965c_43c6_aacb_9ba92b12d4a9.slice/crio-699a453f6cb66184098a1202c061496f5f2ddb18deb913031ea7867a56fd5a5c WatchSource:0}: Error finding container 699a453f6cb66184098a1202c061496f5f2ddb18deb913031ea7867a56fd5a5c: Status 404 returned error can't find the container with id 699a453f6cb66184098a1202c061496f5f2ddb18deb913031ea7867a56fd5a5c
Jul 28 12:53:22.661200 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.661166    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:22.661354 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.661215    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:22.661354 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:22.661248    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:22.661354 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.661318    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:22.661354 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.661335    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:22.661354 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.661343    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:22.661597 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.661365    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:22.661597 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.661397    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs podName:43fa5143-75d6-4f79-88d0-97b5a5ecf09c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:23.66137972 +0000 UTC m=+2.984372658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs") pod "network-metrics-daemon-k4prs" (UID: "43fa5143-75d6-4f79-88d0-97b5a5ecf09c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:22.661597 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.661416    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret podName:64386eb9-8899-410e-adf0-3f21a2a226b0 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:23.661406429 +0000 UTC m=+2.984399353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret") pod "global-pull-secret-syncer-p6smg" (UID: "64386eb9-8899-410e-adf0-3f21a2a226b0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:22.661597 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.661344    2572 projected.go:196] Error preparing data for projected volume kube-api-access-rt77l for pod openshift-network-diagnostics/network-check-target-lkksh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:22.661597 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:22.661457    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l podName:64cdd13d-520d-4f12-86fa-a5f99d362fba nodeName:}" failed. No retries permitted until 2026-07-28 12:53:23.661446425 +0000 UTC m=+2.984439343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rt77l" (UniqueName: "kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l") pod "network-check-target-lkksh" (UID: "64cdd13d-520d-4f12-86fa-a5f99d362fba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:23.091428 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.091287    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:48:22 +0000 UTC" deadline="2028-01-02 00:24:06.851641003 +0000 UTC"
Jul 28 12:53:23.091428 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.091331    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12539h30m43.760315846s"
Jul 28 12:53:23.177876 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.177412    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:23.177876 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:23.177529    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:23.192481 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.192250    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwszv" event={"ID":"f6b221b1-965c-43c6-aacb-9ba92b12d4a9","Type":"ContainerStarted","Data":"699a453f6cb66184098a1202c061496f5f2ddb18deb913031ea7867a56fd5a5c"}
Jul 28 12:53:23.214280 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.214190    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tvzl9" event={"ID":"11b94fe7-fd89-41af-b09b-79cfcb8afad8","Type":"ContainerStarted","Data":"dbe84bc6b10d3fe8b8fe6c8d314bae0f27d1ae0092a781620750e0406a2c2f5c"}
Jul 28 12:53:23.218675 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.218637    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7rl6" event={"ID":"b52c3e58-14e8-442c-b277-6ba3718e5682","Type":"ContainerStarted","Data":"48daf7380bf0f8e1407376ba9019fa32d9b051817bf69087e13d87d1f217d5a1"}
Jul 28 12:53:23.223968 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.223919    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal" event={"ID":"5e36d7aafc98cad33ea701597790b088","Type":"ContainerStarted","Data":"eb773264f8b392f0989f2dcbd7ecc23bc26eb01827d2ff15048064ecfcf8b2b2"}
Jul 28 12:53:23.235778 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.235741    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" event={"ID":"66460849-0183-481b-a998-ac5bc9c81e35","Type":"ContainerStarted","Data":"66e7bcccb5d2daed4f14e8c3a698884a81e47ab9658502166e90aad027a56c7f"}
Jul 28 12:53:23.250304 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.250206    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mpcpt" event={"ID":"bd90f16b-a738-49f8-99f8-ac9981959493","Type":"ContainerStarted","Data":"bb6f506956b1c361d6121366e54220125728ff746f0de410fbb6b185352f6e06"}
Jul 28 12:53:23.264130 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.264092    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-stdct" event={"ID":"8581eb58-b73d-4cc3-983c-585bb739933c","Type":"ContainerStarted","Data":"a7224b21e9dcaef593d687289cc2241cad3efd23a084b81619970517f6c7c47d"}
Jul 28 12:53:23.275401 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.275364    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ksgqt" event={"ID":"c0e02d46-2f01-4d45-b048-f0dc22cfad61","Type":"ContainerStarted","Data":"f7ebe1346e4075eb8dedd9a324cd96e90ed168ae323ede7083bf3551b64089df"}
Jul 28 12:53:23.281228 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.281192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-szhdp" event={"ID":"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96","Type":"ContainerStarted","Data":"cc60d5e8f206ca43fd03d5dae687eccb4bd22d091380ac2448cd4eb14f9c3bf8"}
Jul 28 12:53:23.292013 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.291974    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4" event={"ID":"cbe8cb75-1a5f-4927-914a-e7e7c7c80350","Type":"ContainerStarted","Data":"3f3b08893fb2f31fe17e9e8910b0660dab826b36d2bf469f40a3b8bb3d92769b"}
Jul 28 12:53:23.302457 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.302419    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal" event={"ID":"755ebb668ab847a4c2017edcdd11706f","Type":"ContainerStarted","Data":"2b5bc4ccb9cacb2de48efc515186c5653bb8222f7ddc833585fcb15350dd7018"}
Jul 28 12:53:23.429278 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.429201    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:23.675613 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.675446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:23.675613 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.675505    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:23.675613 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:23.675557    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:23.676039 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:23.675704    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:23.676039 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:23.675728    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:23.676039 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:23.675742    2572 projected.go:196] Error preparing data for projected volume kube-api-access-rt77l for pod openshift-network-diagnostics/network-check-target-lkksh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:23.676039 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:23.675798    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l podName:64cdd13d-520d-4f12-86fa-a5f99d362fba nodeName:}" failed. No retries permitted until 2026-07-28 12:53:25.675780188 +0000 UTC m=+4.998773120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rt77l" (UniqueName: "kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l") pod "network-check-target-lkksh" (UID: "64cdd13d-520d-4f12-86fa-a5f99d362fba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:23.676310 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:23.676218    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:23.676310 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:23.676270    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret podName:64386eb9-8899-410e-adf0-3f21a2a226b0 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:25.676254408 +0000 UTC m=+4.999247329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret") pod "global-pull-secret-syncer-p6smg" (UID: "64386eb9-8899-410e-adf0-3f21a2a226b0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:23.676426 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:23.676328    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:23.676426 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:23.676365    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs podName:43fa5143-75d6-4f79-88d0-97b5a5ecf09c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:25.676352187 +0000 UTC m=+4.999345109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs") pod "network-metrics-daemon-k4prs" (UID: "43fa5143-75d6-4f79-88d0-97b5a5ecf09c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:24.092230 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:24.092165    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:48:22 +0000 UTC" deadline="2028-05-06 05:41:19.32076222 +0000 UTC"
Jul 28 12:53:24.092230 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:24.092229    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15544h47m55.22853789s"
Jul 28 12:53:24.174972 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:24.174926    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:24.175157 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:24.175068    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:24.175497 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:24.175475    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:24.175612 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:24.175592    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:25.177965 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:25.177887    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:25.178386 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:25.178035    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:25.693124 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:25.693084    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:25.693288 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:25.693152    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:25.693288 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:25.693201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:25.693377 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:25.693355    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:25.693413 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:25.693372    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:25.693413 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:25.693395    2572 projected.go:196] Error preparing data for projected volume kube-api-access-rt77l for pod openshift-network-diagnostics/network-check-target-lkksh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:25.693477 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:25.693448    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l podName:64cdd13d-520d-4f12-86fa-a5f99d362fba nodeName:}" failed. No retries permitted until 2026-07-28 12:53:29.69343198 +0000 UTC m=+9.016424903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rt77l" (UniqueName: "kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l") pod "network-check-target-lkksh" (UID: "64cdd13d-520d-4f12-86fa-a5f99d362fba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:25.693900 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:25.693868    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:25.693987 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:25.693918    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret podName:64386eb9-8899-410e-adf0-3f21a2a226b0 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:29.693903287 +0000 UTC m=+9.016896207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret") pod "global-pull-secret-syncer-p6smg" (UID: "64386eb9-8899-410e-adf0-3f21a2a226b0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:25.694034 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:25.693994    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:25.694034 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:25.694029    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs podName:43fa5143-75d6-4f79-88d0-97b5a5ecf09c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:29.694015524 +0000 UTC m=+9.017008448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs") pod "network-metrics-daemon-k4prs" (UID: "43fa5143-75d6-4f79-88d0-97b5a5ecf09c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:26.176063 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:26.175393    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:26.176063 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:26.175527    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:26.176063 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:26.175907    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:26.176063 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:26.176026    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:27.177359 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:27.177301    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:27.177757 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:27.177436    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:28.175545 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:28.175511    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:28.175717 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:28.175644    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:28.176023 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:28.175996    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:28.176163 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:28.176085    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:29.175590 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:29.175549    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:29.176085 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:29.175691    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:29.726674    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:29.726758    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:29.726807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:29.726955    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:29.727022    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret podName:64386eb9-8899-410e-adf0-3f21a2a226b0 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:37.727002706 +0000 UTC m=+17.049995627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret") pod "global-pull-secret-syncer-p6smg" (UID: "64386eb9-8899-410e-adf0-3f21a2a226b0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:29.727421    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:29.727475    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs podName:43fa5143-75d6-4f79-88d0-97b5a5ecf09c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:37.72745746 +0000 UTC m=+17.050450381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs") pod "network-metrics-daemon-k4prs" (UID: "43fa5143-75d6-4f79-88d0-97b5a5ecf09c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:29.727555    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:29.727572    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:29.727585    2572 projected.go:196] Error preparing data for projected volume kube-api-access-rt77l for pod openshift-network-diagnostics/network-check-target-lkksh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:29.727659 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:29.727619    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l podName:64cdd13d-520d-4f12-86fa-a5f99d362fba nodeName:}" failed. No retries permitted until 2026-07-28 12:53:37.727608069 +0000 UTC m=+17.050600994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rt77l" (UniqueName: "kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l") pod "network-check-target-lkksh" (UID: "64cdd13d-520d-4f12-86fa-a5f99d362fba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:30.175507 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:30.175419    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:30.175507 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:30.175466    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:30.176078 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:30.175594    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:30.176078 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:30.176026    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:31.175665 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:31.175627    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:31.176165 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:31.175746    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:32.175693 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:32.175116    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:32.175693 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:32.175227    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:32.175693 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:32.175586    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:32.175693 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:32.175655    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:33.174639 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:33.174604    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:33.174809 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:33.174741    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:34.174507 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:34.174473    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:34.174507 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:34.174512    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:34.174918 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:34.174595    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:34.174918 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:34.174709    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:35.175144 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:35.175110    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:35.175535 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:35.175228    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:36.175368 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:36.175088    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:36.175368 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:36.175118    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:36.175368 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:36.175228    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:36.175368 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:36.175363    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:37.175241 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:37.175203    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:37.175411 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:37.175323    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:37.792936 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:37.792902    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:37.793118 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:37.792972    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:37.793118 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:37.793009    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:37.793118 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:37.793070    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:37.793118 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:37.793093    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:37.793118 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:37.793095    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:37.793118 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:37.793103    2572 projected.go:196] Error preparing data for projected volume kube-api-access-rt77l for pod openshift-network-diagnostics/network-check-target-lkksh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:37.793437 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:37.793141    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs podName:43fa5143-75d6-4f79-88d0-97b5a5ecf09c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:53.793123784 +0000 UTC m=+33.116116702 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs") pod "network-metrics-daemon-k4prs" (UID: "43fa5143-75d6-4f79-88d0-97b5a5ecf09c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:37.793437 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:37.793155    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l podName:64cdd13d-520d-4f12-86fa-a5f99d362fba nodeName:}" failed. No retries permitted until 2026-07-28 12:53:53.79314886 +0000 UTC m=+33.116141777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rt77l" (UniqueName: "kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l") pod "network-check-target-lkksh" (UID: "64cdd13d-520d-4f12-86fa-a5f99d362fba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:37.793437 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:37.793169    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:37.793437 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:37.793204    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret podName:64386eb9-8899-410e-adf0-3f21a2a226b0 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:53.793187782 +0000 UTC m=+33.116180700 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret") pod "global-pull-secret-syncer-p6smg" (UID: "64386eb9-8899-410e-adf0-3f21a2a226b0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:38.174899 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:38.174830    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:38.175033 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:38.174830    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:38.175033 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:38.174965    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:38.175033 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:38.175020    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:39.175410 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:39.175373    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:39.175855 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:39.175515    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:40.175062 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:40.175024    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:40.175232 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:40.175025    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:40.175232 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:40.175156    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:40.175362 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:40.175223    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:41.175485 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:41.175445    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:41.175984 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:41.175574    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:42.175347 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:42.175316    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:42.175347 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:42.175351    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:42.175882 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:42.175447    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:42.175882 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:42.175547    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:43.175014 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.174830    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:43.175177 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:43.175144    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:43.346712 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.346680    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-stdct" event={"ID":"8581eb58-b73d-4cc3-983c-585bb739933c","Type":"ContainerStarted","Data":"3464769ab2caa23154b49fd9842e11d085a02378de8627196f4799448c6403e3"}
Jul 28 12:53:43.348168 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.348139    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ksgqt" event={"ID":"c0e02d46-2f01-4d45-b048-f0dc22cfad61","Type":"ContainerStarted","Data":"eee25404de87572d62b76d9d5b27b8d223d97c9d8221db5ac2ed6c53d50a10d6"}
Jul 28 12:53:43.349469 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.349446    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-szhdp" event={"ID":"ad991f88-1a5a-4745-b3c6-7f00b9f4ee96","Type":"ContainerStarted","Data":"dfb6dd5eb14d629d22cbc2bd0b135a314f3e5efd6502e30da75c33575598aca7"}
Jul 28 12:53:43.350873 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.350852    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4" event={"ID":"cbe8cb75-1a5f-4927-914a-e7e7c7c80350","Type":"ContainerStarted","Data":"4a32c0c404873e80138e8871ccee797b9c5ac1189cdff2cdd93ce11194665021"}
Jul 28 12:53:43.352212 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.352193    2572 generic.go:356] "Generic (PLEG): container finished" podID="755ebb668ab847a4c2017edcdd11706f" containerID="0adfd70cc1095391e8f44909f36382ca713b99c1f369ba5422609d1da990e5d9" exitCode=0
Jul 28 12:53:43.352296 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.352256    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal" event={"ID":"755ebb668ab847a4c2017edcdd11706f","Type":"ContainerDied","Data":"0adfd70cc1095391e8f44909f36382ca713b99c1f369ba5422609d1da990e5d9"}
Jul 28 12:53:43.353539 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.353520    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwszv" event={"ID":"f6b221b1-965c-43c6-aacb-9ba92b12d4a9","Type":"ContainerStarted","Data":"6e0d7c2ac458d79ee405e92992f169dd7e3c4552fac901fe4d939f465145ee09"}
Jul 28 12:53:43.354745 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.354726    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7rl6" event={"ID":"b52c3e58-14e8-442c-b277-6ba3718e5682","Type":"ContainerStarted","Data":"c7d3bad9d613a2546c63bba8ee9ff1f3a54501235e7eaa319cd48c29a2ecad93"}
Jul 28 12:53:43.356059 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.356040    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal" event={"ID":"5e36d7aafc98cad33ea701597790b088","Type":"ContainerStarted","Data":"e89b2415b092aba794424cd1b8a6fbb7e9c8b0234d7853bf85025154ca8e7f79"}
Jul 28 12:53:43.358299 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.358281    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" event={"ID":"66460849-0183-481b-a998-ac5bc9c81e35","Type":"ContainerStarted","Data":"f695bf91151159ce3303844b9e85535ff1afea826107f3e510bdd9f33894ef0f"}
Jul 28 12:53:43.358377 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.358331    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" event={"ID":"66460849-0183-481b-a998-ac5bc9c81e35","Type":"ContainerStarted","Data":"df26f85f776fb87295db9403230dba445c2b951687d81eabc372c8735aa69ea4"}
Jul 28 12:53:43.358377 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.358360    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" event={"ID":"66460849-0183-481b-a998-ac5bc9c81e35","Type":"ContainerStarted","Data":"cd8e90bd0192f656f4786e00e84eac7848442282da93efccea04eca00a1b051a"}
Jul 28 12:53:43.358454 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.358420    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" event={"ID":"66460849-0183-481b-a998-ac5bc9c81e35","Type":"ContainerStarted","Data":"8af5a8894b86588faf5f73b055cf0fcc660356413be9f7c92e381f3caae8a1cd"}
Jul 28 12:53:43.359769 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.359723    2572 generic.go:356] "Generic (PLEG): container finished" podID="bd90f16b-a738-49f8-99f8-ac9981959493" containerID="5a7cec141bacd06259e5856a7f7115348b9dc79ba37f0566f9d1f6df67e8c79f" exitCode=0
Jul 28 12:53:43.359857 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.359775    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mpcpt" event={"ID":"bd90f16b-a738-49f8-99f8-ac9981959493","Type":"ContainerDied","Data":"5a7cec141bacd06259e5856a7f7115348b9dc79ba37f0566f9d1f6df67e8c79f"}
Jul 28 12:53:43.366651 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.366606    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-stdct" podStartSLOduration=2.25870701 podStartE2EDuration="22.366588659s" totalImagesPullingTime="20.107881649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:43.366467935 +0000 UTC m=+22.689460876" watchObservedRunningTime="2026-07-28 12:53:43.366588659 +0000 UTC m=+22.689581598"
Jul 28 12:53:43.384272 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.384228    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bwszv" podStartSLOduration=2.447043405 podStartE2EDuration="22.384211539s" totalImagesPullingTime="19.937168134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:43.384164568 +0000 UTC m=+22.707157511" watchObservedRunningTime="2026-07-28 12:53:43.384211539 +0000 UTC m=+22.707204480"
Jul 28 12:53:43.401761 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.401717    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-83.ec2.internal" podStartSLOduration=22.401701493 podStartE2EDuration="22.401701493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:43.398677275 +0000 UTC m=+22.721670215" watchObservedRunningTime="2026-07-28 12:53:43.401701493 +0000 UTC m=+22.724694432"
Jul 28 12:53:43.430223 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.430028    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q7rl6" podStartSLOduration=2.236158889 podStartE2EDuration="22.430011986s" totalImagesPullingTime="20.193853097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:43.429842715 +0000 UTC m=+22.752835653" watchObservedRunningTime="2026-07-28 12:53:43.430011986 +0000 UTC m=+22.753004926"
Jul 28 12:53:43.447067 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.447031    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-szhdp" podStartSLOduration=2.319749497 podStartE2EDuration="22.447016733s" totalImagesPullingTime="20.127267236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:43.446802232 +0000 UTC m=+22.769795187" watchObservedRunningTime="2026-07-28 12:53:43.447016733 +0000 UTC m=+22.770009672"
Jul 28 12:53:43.465057 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:43.464884    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ksgqt" podStartSLOduration=2.328761073 podStartE2EDuration="22.464866057s" totalImagesPullingTime="20.136104984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:43.464728012 +0000 UTC m=+22.787720939" watchObservedRunningTime="2026-07-28 12:53:43.464866057 +0000 UTC m=+22.787858998"
Jul 28 12:53:44.175394 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:44.175358    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:44.175558 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:44.175472    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:44.175558 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:44.175501    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:44.175558 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:44.175549    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:44.365013 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:44.364908    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" event={"ID":"66460849-0183-481b-a998-ac5bc9c81e35","Type":"ContainerStarted","Data":"8dd70b98ac5d3d5d0085fbf524f4830e385c2d346e1721fc32fc4b433bcfc8f7"}
Jul 28 12:53:44.365438 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:44.365010    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" event={"ID":"66460849-0183-481b-a998-ac5bc9c81e35","Type":"ContainerStarted","Data":"5fe3052e93122d9c3dff8381806af6298706310419fda13b1ecff51b6c71a030"}
Jul 28 12:53:44.366346 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:44.366314    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tvzl9" event={"ID":"11b94fe7-fd89-41af-b09b-79cfcb8afad8","Type":"ContainerStarted","Data":"9c7a6fd7dc83f2f80c5fcc47d4213198d65087d9b59e128022009682edcae018"}
Jul 28 12:53:44.380657 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:44.380619    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tvzl9" podStartSLOduration=3.2818687779999998 podStartE2EDuration="23.380605642s" totalImagesPullingTime="20.098736864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:44.380366654 +0000 UTC m=+23.703359595" watchObservedRunningTime="2026-07-28 12:53:44.380605642 +0000 UTC m=+23.703598581"
Jul 28 12:53:44.644583 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:44.644556    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:53:45.117685 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:45.117588    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:53:44.644575237Z","UUID":"c6d3141c-a3b8-42b4-a36b-d699efa21a90","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:53:45.119415 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:45.119371    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:53:45.119415 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:45.119422    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:53:45.175580 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:45.175547    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:45.175823 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:45.175684    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:45.370469 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:45.370375    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4" event={"ID":"cbe8cb75-1a5f-4927-914a-e7e7c7c80350","Type":"ContainerStarted","Data":"dab3836b7e95592c436308dbdffa711494e281a78ec0e83527416411262d6d83"}
Jul 28 12:53:45.372019 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:45.371994    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal" event={"ID":"755ebb668ab847a4c2017edcdd11706f","Type":"ContainerStarted","Data":"a695ee7b2dfbe834bea0c58db219899a6c90fff3dbfad7df020db0cf7f9655fc"}
Jul 28 12:53:45.388899 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:45.388849    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-83.ec2.internal" podStartSLOduration=24.388836779000002 podStartE2EDuration="24.388836779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:45.388266977 +0000 UTC m=+24.711259918" watchObservedRunningTime="2026-07-28 12:53:45.388836779 +0000 UTC m=+24.711829719"
Jul 28 12:53:45.771478 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:45.771436    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:45.772033 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:45.772011    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:46.175398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:46.175362    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:46.175398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:46.175402    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:46.175651 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:46.175511    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:46.175651 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:46.175628    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:46.378261 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:46.378224    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" event={"ID":"66460849-0183-481b-a998-ac5bc9c81e35","Type":"ContainerStarted","Data":"05bc8414d67a9a6a99e5025ad14ad75982c24b6073766ceba0d5d440d3c99fa2"}
Jul 28 12:53:46.378748 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:46.378322    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:46.379016 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:46.378998    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q7rl6"
Jul 28 12:53:47.178928 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:47.178900    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:47.179122 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:47.179030    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:47.382440 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:47.382405    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4" event={"ID":"cbe8cb75-1a5f-4927-914a-e7e7c7c80350","Type":"ContainerStarted","Data":"2074f9eb4da57dc23baaebf9e1cf55c209ce5ef5d92899fb53f523584d3b5552"}
Jul 28 12:53:47.403428 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:47.403368    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nsd4" podStartSLOduration=3.019358312 podStartE2EDuration="26.403347633s" totalImagesPullingTime="23.383989321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:47.402797318 +0000 UTC m=+26.725790257" watchObservedRunningTime="2026-07-28 12:53:47.403347633 +0000 UTC m=+26.726340576"
Jul 28 12:53:48.175659 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:48.175395    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:48.175822 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:48.175395    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:48.175822 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:48.175720    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:48.175822 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:48.175748    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:49.178508 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:49.178481    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:49.178875 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:49.178588    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:49.391477 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:49.390870    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" event={"ID":"66460849-0183-481b-a998-ac5bc9c81e35","Type":"ContainerStarted","Data":"488833facf9b9e12d042ae75f2c3744fa5f88d638fc3f29bd8964124910afcc0"}
Jul 28 12:53:49.391477 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:49.391148    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:49.391477 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:49.391213    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:49.408038 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:49.407744    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:49.432091 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:49.432007    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx" podStartSLOduration=8.084472934 podStartE2EDuration="28.431992215s" totalImagesPullingTime="20.347519281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:49.431632719 +0000 UTC m=+28.754625714" watchObservedRunningTime="2026-07-28 12:53:49.431992215 +0000 UTC m=+28.754985154"
Jul 28 12:53:50.175464 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:50.175233    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:50.175729 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:50.175233    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:50.175729 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:50.175505    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:50.175729 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:50.175558    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:50.394345 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:50.394316    2572 generic.go:356] "Generic (PLEG): container finished" podID="bd90f16b-a738-49f8-99f8-ac9981959493" containerID="1cb791d2dc3a5949037dc31607fa5148903d2890aa8727ca9ef863e3628976bd" exitCode=0
Jul 28 12:53:50.395192 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:50.394403    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mpcpt" event={"ID":"bd90f16b-a738-49f8-99f8-ac9981959493","Type":"ContainerDied","Data":"1cb791d2dc3a5949037dc31607fa5148903d2890aa8727ca9ef863e3628976bd"}
Jul 28 12:53:50.395192 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:50.394745    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:50.409794 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:50.409772    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:53:51.175902 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:51.175866    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:51.176077 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:51.176007    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:51.354333 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:51.354305    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k4prs"]
Jul 28 12:53:51.354508 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:51.354426    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:51.354551 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:51.354518    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:51.357642 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:51.357614    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lkksh"]
Jul 28 12:53:51.357748 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:51.357709    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:51.357814 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:51.357788    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:51.358245 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:51.358224    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p6smg"]
Jul 28 12:53:51.396120 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:51.396093    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:51.396455 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:51.396190    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:53.177934 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:53.177857    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:53.178353 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:53.177967    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:53.178353 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.177977    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:53.178353 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:53.177997    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:53.178353 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.178098    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:53.178353 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.178181    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:53.402964 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:53.402912    2572 generic.go:356] "Generic (PLEG): container finished" podID="bd90f16b-a738-49f8-99f8-ac9981959493" containerID="d3f4ca3b5adec60b154238ae576b47ff97c83d2a7c8d6c9f23300cd2765f3318" exitCode=0
Jul 28 12:53:53.403122 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:53.402973    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mpcpt" event={"ID":"bd90f16b-a738-49f8-99f8-ac9981959493","Type":"ContainerDied","Data":"d3f4ca3b5adec60b154238ae576b47ff97c83d2a7c8d6c9f23300cd2765f3318"}
Jul 28 12:53:53.812414 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:53.812374    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:53.812414 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:53.812424    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:53.812687 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.812521    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:53.812687 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.812558    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:53.812687 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.812584    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:53.812687 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.812596    2572 projected.go:196] Error preparing data for projected volume kube-api-access-rt77l for pod openshift-network-diagnostics/network-check-target-lkksh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:53.812687 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.812568    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret podName:64386eb9-8899-410e-adf0-3f21a2a226b0 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:25.812556184 +0000 UTC m=+65.135549102 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret") pod "global-pull-secret-syncer-p6smg" (UID: "64386eb9-8899-410e-adf0-3f21a2a226b0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:53.812687 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:53.812630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:53.812687 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.812648    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l podName:64cdd13d-520d-4f12-86fa-a5f99d362fba nodeName:}" failed. No retries permitted until 2026-07-28 12:54:25.812632302 +0000 UTC m=+65.135625231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rt77l" (UniqueName: "kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l") pod "network-check-target-lkksh" (UID: "64cdd13d-520d-4f12-86fa-a5f99d362fba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:53.812917 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.812695    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:53.812917 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:53.812737    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs podName:43fa5143-75d6-4f79-88d0-97b5a5ecf09c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:25.81272548 +0000 UTC m=+65.135718403 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs") pod "network-metrics-daemon-k4prs" (UID: "43fa5143-75d6-4f79-88d0-97b5a5ecf09c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:55.174542 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.174511    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:55.174542 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.174539    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:55.175029 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.174512    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:55.175029 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:55.174619    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p6smg" podUID="64386eb9-8899-410e-adf0-3f21a2a226b0"
Jul 28 12:53:55.175029 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:55.174691    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkksh" podUID="64cdd13d-520d-4f12-86fa-a5f99d362fba"
Jul 28 12:53:55.175029 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:55.174801    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4prs" podUID="43fa5143-75d6-4f79-88d0-97b5a5ecf09c"
Jul 28 12:53:55.409374 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.409339    2572 generic.go:356] "Generic (PLEG): container finished" podID="bd90f16b-a738-49f8-99f8-ac9981959493" containerID="4a6fc9af12d6ba060c558a9924ef729c52766b47c60133d1aab9504ce951e2a2" exitCode=0
Jul 28 12:53:55.409569 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.409382    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mpcpt" event={"ID":"bd90f16b-a738-49f8-99f8-ac9981959493","Type":"ContainerDied","Data":"4a6fc9af12d6ba060c558a9924ef729c52766b47c60133d1aab9504ce951e2a2"}
Jul 28 12:53:55.470746 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.470671    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-83.ec2.internal" event="NodeReady"
Jul 28 12:53:55.470908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.470794    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:53:55.507984 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.507937    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b4bf4b95c-pzrf9"]
Jul 28 12:53:55.511444 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.511430    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.513509 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.513487    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 12:53:55.513591 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.513538    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 12:53:55.513632 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.513540    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fzfhc\""
Jul 28 12:53:55.513826 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.513809    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 12:53:55.519433 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.519410    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 12:53:55.523012 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.522991    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4bf4b95c-pzrf9"]
Jul 28 12:53:55.527623 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.527604    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5s4dp"]
Jul 28 12:53:55.531434 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.531419    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5c2st"]
Jul 28 12:53:55.531547 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.531530    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5s4dp"
Jul 28 12:53:55.533780 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.533760    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:53:55.533866 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.533768    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:53:55.533866 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.533832    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gt9sx\""
Jul 28 12:53:55.533866 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.533861    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:53:55.535308 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.535292    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.536953 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.536924    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:53:55.537188 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.537072    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:53:55.537188 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.537139    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xjd7z\""
Jul 28 12:53:55.540144 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.540124    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5s4dp"]
Jul 28 12:53:55.546716 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.546693    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c2st"]
Jul 28 12:53:55.623513 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623475    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-trusted-ca\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.623513 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5a807c-9e19-4693-acc3-20e97affcb82-tmp-dir\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.623726 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623542    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0435f7c0-0018-49a2-b8c1-c4048884fc03-ca-trust-extracted\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.623726 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.623726 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623619    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-installation-pull-secrets\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.623726 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623659    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5a807c-9e19-4693-acc3-20e97affcb82-config-volume\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.623726 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623686    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hs5\" (UniqueName: \"kubernetes.io/projected/7f5a807c-9e19-4693-acc3-20e97affcb82-kube-api-access-q5hs5\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.623726 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-image-registry-private-configuration\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.623934 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-certificates\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.623934 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623770    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4znz\" (UniqueName: \"kubernetes.io/projected/c1ea0458-166f-4cab-a151-ae16ad59c313-kube-api-access-l4znz\") pod \"ingress-canary-5s4dp\" (UID: \"c1ea0458-166f-4cab-a151-ae16ad59c313\") " pod="openshift-ingress-canary/ingress-canary-5s4dp"
Jul 28 12:53:55.623934 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623791    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.623934 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623823    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-bound-sa-token\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.623934 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623842    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1ea0458-166f-4cab-a151-ae16ad59c313-cert\") pod \"ingress-canary-5s4dp\" (UID: \"c1ea0458-166f-4cab-a151-ae16ad59c313\") " pod="openshift-ingress-canary/ingress-canary-5s4dp"
Jul 28 12:53:55.623934 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.623856    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48spq\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-kube-api-access-48spq\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.689267 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.689237    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:53:55.724233 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-certificates\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.724233 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4znz\" (UniqueName: \"kubernetes.io/projected/c1ea0458-166f-4cab-a151-ae16ad59c313-kube-api-access-l4znz\") pod \"ingress-canary-5s4dp\" (UID: \"c1ea0458-166f-4cab-a151-ae16ad59c313\") " pod="openshift-ingress-canary/ingress-canary-5s4dp"
Jul 28 12:53:55.724233 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724220    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724249    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-bound-sa-token\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1ea0458-166f-4cab-a151-ae16ad59c313-cert\") pod \"ingress-canary-5s4dp\" (UID: \"c1ea0458-166f-4cab-a151-ae16ad59c313\") " pod="openshift-ingress-canary/ingress-canary-5s4dp"
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724289    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48spq\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-kube-api-access-48spq\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724327    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-trusted-ca\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5a807c-9e19-4693-acc3-20e97affcb82-tmp-dir\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724364    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0435f7c0-0018-49a2-b8c1-c4048884fc03-ca-trust-extracted\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:55.724365    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724378    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:55.724386    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4bf4b95c-pzrf9: secret "image-registry-tls" not found
Jul 28 12:53:55.724398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-installation-pull-secrets\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.724862 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724417    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5a807c-9e19-4693-acc3-20e97affcb82-config-volume\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.724862 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:55.724441    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls podName:0435f7c0-0018-49a2-b8c1-c4048884fc03 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:56.224422091 +0000 UTC m=+35.547415008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls") pod "image-registry-b4bf4b95c-pzrf9" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03") : secret "image-registry-tls" not found
Jul 28 12:53:55.724862 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724474    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hs5\" (UniqueName: \"kubernetes.io/projected/7f5a807c-9e19-4693-acc3-20e97affcb82-kube-api-access-q5hs5\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.724862 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724515    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-image-registry-private-configuration\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.724862 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724822    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0435f7c0-0018-49a2-b8c1-c4048884fc03-ca-trust-extracted\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.725135 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:55.724877    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:55.725135 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.724909    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-certificates\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.725135 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:55.724929    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls podName:7f5a807c-9e19-4693-acc3-20e97affcb82 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:56.224912346 +0000 UTC m=+35.547905278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls") pod "dns-default-5c2st" (UID: "7f5a807c-9e19-4693-acc3-20e97affcb82") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:55.725135 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.725098    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5a807c-9e19-4693-acc3-20e97affcb82-tmp-dir\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.725135 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.725127    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5a807c-9e19-4693-acc3-20e97affcb82-config-volume\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.725599 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.725576    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-trusted-ca\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.728753 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.728726    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1ea0458-166f-4cab-a151-ae16ad59c313-cert\") pod \"ingress-canary-5s4dp\" (UID: \"c1ea0458-166f-4cab-a151-ae16ad59c313\") " pod="openshift-ingress-canary/ingress-canary-5s4dp"
Jul 28 12:53:55.728753 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.728736    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-installation-pull-secrets\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.728914 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.728741    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-image-registry-private-configuration\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.738639 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.734465    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4znz\" (UniqueName: \"kubernetes.io/projected/c1ea0458-166f-4cab-a151-ae16ad59c313-kube-api-access-l4znz\") pod \"ingress-canary-5s4dp\" (UID: \"c1ea0458-166f-4cab-a151-ae16ad59c313\") " pod="openshift-ingress-canary/ingress-canary-5s4dp"
Jul 28 12:53:55.739322 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.739297    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48spq\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-kube-api-access-48spq\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.740058 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.740033    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hs5\" (UniqueName: \"kubernetes.io/projected/7f5a807c-9e19-4693-acc3-20e97affcb82-kube-api-access-q5hs5\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:55.743908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.743883    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-bound-sa-token\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:55.815082 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.815040    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:53:55.841507 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.841474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5s4dp"
Jul 28 12:53:55.999060 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:55.998973    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5s4dp"]
Jul 28 12:53:56.005274 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:53:56.005241    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ea0458_166f_4cab_a151_ae16ad59c313.slice/crio-7e8157d29e36457d70db39ef9d4423088be221dfec3ce652eef8376342a16668 WatchSource:0}: Error finding container 7e8157d29e36457d70db39ef9d4423088be221dfec3ce652eef8376342a16668: Status 404 returned error can't find the container with id 7e8157d29e36457d70db39ef9d4423088be221dfec3ce652eef8376342a16668
Jul 28 12:53:56.229760 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:56.229505    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:56.230594 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:56.229818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:56.230594 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:56.229698    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:56.230594 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:56.229888    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4bf4b95c-pzrf9: secret "image-registry-tls" not found
Jul 28 12:53:56.230594 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:56.229907    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:56.230594 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:56.229963    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls podName:0435f7c0-0018-49a2-b8c1-c4048884fc03 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:57.229926717 +0000 UTC m=+36.552919636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls") pod "image-registry-b4bf4b95c-pzrf9" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03") : secret "image-registry-tls" not found
Jul 28 12:53:56.230594 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:56.229988    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls podName:7f5a807c-9e19-4693-acc3-20e97affcb82 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:57.229970156 +0000 UTC m=+36.552963085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls") pod "dns-default-5c2st" (UID: "7f5a807c-9e19-4693-acc3-20e97affcb82") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:56.413477 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:56.413387    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5s4dp" event={"ID":"c1ea0458-166f-4cab-a151-ae16ad59c313","Type":"ContainerStarted","Data":"7e8157d29e36457d70db39ef9d4423088be221dfec3ce652eef8376342a16668"}
Jul 28 12:53:57.175023 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.174983    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:53:57.175208 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.174983    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:53:57.175438 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.175422    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:53:57.178103 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.178074    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:53:57.178246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.178096    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:53:57.178246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.178130    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z7m8f\""
Jul 28 12:53:57.178246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.178137    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kmztt\""
Jul 28 12:53:57.178246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.178080    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:53:57.178246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.178080    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:53:57.240130 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.240096    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:57.240704 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:57.240183    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:57.240704 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:57.240295    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:57.240704 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:57.240321    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:57.240704 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:57.240335    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4bf4b95c-pzrf9: secret "image-registry-tls" not found
Jul 28 12:53:57.240704 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:57.240384    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls podName:7f5a807c-9e19-4693-acc3-20e97affcb82 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:59.240356922 +0000 UTC m=+38.563349843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls") pod "dns-default-5c2st" (UID: "7f5a807c-9e19-4693-acc3-20e97affcb82") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:57.240704 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:57.240406    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls podName:0435f7c0-0018-49a2-b8c1-c4048884fc03 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:59.240396999 +0000 UTC m=+38.563389917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls") pod "image-registry-b4bf4b95c-pzrf9" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03") : secret "image-registry-tls" not found
Jul 28 12:53:59.256805 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:59.256766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:53:59.257277 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:53:59.256848    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:53:59.257277 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:59.256932    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:59.257277 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:59.256963    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4bf4b95c-pzrf9: secret "image-registry-tls" not found
Jul 28 12:53:59.257277 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:59.256976    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:59.257277 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:59.257028    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls podName:7f5a807c-9e19-4693-acc3-20e97affcb82 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:03.257008505 +0000 UTC m=+42.580001423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls") pod "dns-default-5c2st" (UID: "7f5a807c-9e19-4693-acc3-20e97affcb82") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:59.257277 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:53:59.257045    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls podName:0435f7c0-0018-49a2-b8c1-c4048884fc03 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:03.257038883 +0000 UTC m=+42.580031815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls") pod "image-registry-b4bf4b95c-pzrf9" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03") : secret "image-registry-tls" not found
Jul 28 12:54:00.090147 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:00.090118    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:00.190969 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:00.190924    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:00.425652 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:00.425618    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5s4dp" event={"ID":"c1ea0458-166f-4cab-a151-ae16ad59c313","Type":"ContainerStarted","Data":"8dcb220c4ff22d59010bf0fc3790f6ee5ec5db5543455dca0ca3cc7371306748"}
Jul 28 12:54:00.440597 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:00.440530    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5s4dp" podStartSLOduration=1.159022713 podStartE2EDuration="5.440508552s" totalImagesPullingTime="4.281485839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:00.440110843 +0000 UTC m=+39.763103784" watchObservedRunningTime="2026-07-28 12:54:00.440508552 +0000 UTC m=+39.763501473"
Jul 28 12:54:01.430603 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:01.430564    2572 generic.go:356] "Generic (PLEG): container finished" podID="bd90f16b-a738-49f8-99f8-ac9981959493" containerID="acb167072b061c9052cb1e9f9edde5cb6f03fa1714faed4736693906fc2fdb60" exitCode=0
Jul 28 12:54:01.431058 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:01.430639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mpcpt" event={"ID":"bd90f16b-a738-49f8-99f8-ac9981959493","Type":"ContainerDied","Data":"acb167072b061c9052cb1e9f9edde5cb6f03fa1714faed4736693906fc2fdb60"}
Jul 28 12:54:02.141352 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.141317    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:02.212394 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.212359    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:02.435662 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.435570    2572 generic.go:356] "Generic (PLEG): container finished" podID="bd90f16b-a738-49f8-99f8-ac9981959493" containerID="9412ba23d6e543f772657bf0a2c455f63521a521415396a93a8a06cfcd144d0a" exitCode=0
Jul 28 12:54:02.435662 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.435640    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mpcpt" event={"ID":"bd90f16b-a738-49f8-99f8-ac9981959493","Type":"ContainerDied","Data":"9412ba23d6e543f772657bf0a2c455f63521a521415396a93a8a06cfcd144d0a"}
Jul 28 12:54:02.682455 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.682423    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"]
Jul 28 12:54:02.704603 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.704530    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f"]
Jul 28 12:54:02.704721 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.704656    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:02.706957 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.706926    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 12:54:02.707097 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.707007    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 12:54:02.707097 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.707034    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:54:02.707273 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.707256    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 12:54:02.707528 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.707510    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lspfw\""
Jul 28 12:54:02.725384 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.725363    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-zmt5l"]
Jul 28 12:54:02.725508 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.725487    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f"
Jul 28 12:54:02.727973 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.727932    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:54:02.728081 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.727972    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 12:54:02.728081 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.727932    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7jtlw\""
Jul 28 12:54:02.749561 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.749539    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx"]
Jul 28 12:54:02.749690 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.749672    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.751668 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.751650    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 12:54:02.751883 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.751867    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 12:54:02.752043 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.751868    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-45rbj\""
Jul 28 12:54:02.752043 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.751912    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 12:54:02.752043 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.752012    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:54:02.757479 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.757459    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 12:54:02.773557 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.773534    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx"]
Jul 28 12:54:02.773651 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.773563    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"]
Jul 28 12:54:02.773651 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.773578    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f"]
Jul 28 12:54:02.773651 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.773597    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-zmt5l"]
Jul 28 12:54:02.773651 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.773623    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"]
Jul 28 12:54:02.773651 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.773639    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx"
Jul 28 12:54:02.776417 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.776396    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4x65m\""
Jul 28 12:54:02.793686 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.793651    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"]
Jul 28 12:54:02.793840 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.793824    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:02.798621 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.798580    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 12:54:02.799809 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.798936    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vlcxf\""
Jul 28 12:54:02.802097 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.801015    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 12:54:02.816965 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.811724    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rgl5v"]
Jul 28 12:54:02.823881 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.823853    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:02.826751 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.826730    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 12:54:02.826751 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.826745    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:54:02.826901 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.826756    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kg6wq\""
Jul 28 12:54:02.826901 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.826795    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 12:54:02.841797 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.841775    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8b6d76f97-jh5t9"]
Jul 28 12:54:02.841908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.841887    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:02.843982 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.843966    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:54:02.844055 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.844016    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gglxq\""
Jul 28 12:54:02.844282 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.844265    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 12:54:02.844331 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.844291    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 12:54:02.844686 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.844673    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:54:02.849443 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.849423    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 12:54:02.861952 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.861920    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:02.861952 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.861928    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"]
Jul 28 12:54:02.862108 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.861990    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"]
Jul 28 12:54:02.862108 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.862005    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rgl5v"]
Jul 28 12:54:02.862108 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.862019    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8b6d76f97-jh5t9"]
Jul 28 12:54:02.864062 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.864045    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c2ppd\""
Jul 28 12:54:02.864164 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.864149    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 12:54:02.864296 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.864279    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:54:02.864506 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.864492    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 12:54:02.864582 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.864514    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:54:02.864582 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.864537    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 12:54:02.864654 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.864580    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 12:54:02.876367 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.876344    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"]
Jul 28 12:54:02.886537 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886494    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:02.886637 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886536    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:02.886637 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886562    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb4c7102-dee6-436c-8d1a-3e4b83206308-serving-cert\") pod \"service-ca-operator-5d6c75f84f-45mdv\" (UID: \"cb4c7102-dee6-436c-8d1a-3e4b83206308\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:02.886637 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886591    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vzl\" (UniqueName: \"kubernetes.io/projected/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-kube-api-access-g7vzl\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.886637 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886608    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sxjw\" (UniqueName: \"kubernetes.io/projected/f46daa6b-8762-4b6e-967f-2ae0d994f592-kube-api-access-8sxjw\") pod \"volume-data-source-validator-549c6d54cf-q4x7f\" (UID: \"f46daa6b-8762-4b6e-967f-2ae0d994f592\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f"
Jul 28 12:54:02.886637 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886636    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb4c7102-dee6-436c-8d1a-3e4b83206308-config\") pod \"service-ca-operator-5d6c75f84f-45mdv\" (UID: \"cb4c7102-dee6-436c-8d1a-3e4b83206308\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:02.886819 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886679    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvkzf\" (UniqueName: \"kubernetes.io/projected/cb4c7102-dee6-436c-8d1a-3e4b83206308-kube-api-access-mvkzf\") pod \"service-ca-operator-5d6c75f84f-45mdv\" (UID: \"cb4c7102-dee6-436c-8d1a-3e4b83206308\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:02.886819 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-trusted-ca\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.886819 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-serving-cert\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.886819 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886800    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-config\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.886966 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.886822    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbdgv\" (UniqueName: \"kubernetes.io/projected/b43f2f48-3db6-44ce-a879-e31a97f719d9-kube-api-access-rbdgv\") pod \"network-check-source-7d88cb55d8-gbqpx\" (UID: \"b43f2f48-3db6-44ce-a879-e31a97f719d9\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx"
Jul 28 12:54:02.898078 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.898051    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"]
Jul 28 12:54:02.898188 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.898171    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:02.900952 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.900920    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:54:02.901640 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.901617    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:54:02.901721 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.901626    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 12:54:02.901850 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.901836    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5zjx6\""
Jul 28 12:54:02.901977 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.901963    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 12:54:02.973243 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.973169    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"]
Jul 28 12:54:02.983322 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.983287    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:02.987274 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987256    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb4c7102-dee6-436c-8d1a-3e4b83206308-config\") pod \"service-ca-operator-5d6c75f84f-45mdv\" (UID: \"cb4c7102-dee6-436c-8d1a-3e4b83206308\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:02.987343 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987289    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkh5v\" (UniqueName: \"kubernetes.io/projected/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-kube-api-access-zkh5v\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:02.987343 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987306    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4c4f466e-4360-449f-8ab8-da1732648943-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:02.987450 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987432    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-stats-auth\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:02.987501 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987484    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-config\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.987556 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987520    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-serving-cert\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.987556 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987546    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-default-certificate\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:02.987655 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987574    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:02.987655 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx4st\" (UniqueName: \"kubernetes.io/projected/8f6ec1da-be3a-4780-9199-dc20768d96e2-kube-api-access-jx4st\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:02.987760 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987664    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzk8h\" (UniqueName: \"kubernetes.io/projected/4c4f466e-4360-449f-8ab8-da1732648943-kube-api-access-pzk8h\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:02.987760 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987701    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdgv\" (UniqueName: \"kubernetes.io/projected/b43f2f48-3db6-44ce-a879-e31a97f719d9-kube-api-access-rbdgv\") pod \"network-check-source-7d88cb55d8-gbqpx\" (UID: \"b43f2f48-3db6-44ce-a879-e31a97f719d9\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx"
Jul 28 12:54:02.987760 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987729    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:02.987908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnw99\" (UniqueName: \"kubernetes.io/projected/0d851530-6f45-4c04-96c3-c95bbdf2241e-kube-api-access-cnw99\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:02.987908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987777    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb4c7102-dee6-436c-8d1a-3e4b83206308-config\") pod \"service-ca-operator-5d6c75f84f-45mdv\" (UID: \"cb4c7102-dee6-436c-8d1a-3e4b83206308\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:02.987908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987789    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:02.987908 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:02.987809    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:02.987908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987838    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vzl\" (UniqueName: \"kubernetes.io/projected/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-kube-api-access-g7vzl\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.987908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987868    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6ec1da-be3a-4780-9199-dc20768d96e2-service-ca-bundle\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:02.987908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6ec1da-be3a-4780-9199-dc20768d96e2-serving-cert\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:02.988247 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.987968    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:02.988247 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:02.987977    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert podName:25ff57f5-17a0-46b8-9fa6-6e6a05de1ded nodeName:}" failed. No retries permitted until 2026-07-28 12:54:03.487963981 +0000 UTC m=+42.810956899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9dkd5" (UID: "25ff57f5-17a0-46b8-9fa6-6e6a05de1ded") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:02.988247 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988055    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6ec1da-be3a-4780-9199-dc20768d96e2-tmp\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:02.988247 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvkzf\" (UniqueName: \"kubernetes.io/projected/cb4c7102-dee6-436c-8d1a-3e4b83206308-kube-api-access-mvkzf\") pod \"service-ca-operator-5d6c75f84f-45mdv\" (UID: \"cb4c7102-dee6-436c-8d1a-3e4b83206308\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:02.988247 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988125    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f6ec1da-be3a-4780-9199-dc20768d96e2-snapshots\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:02.988247 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988151    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6ec1da-be3a-4780-9199-dc20768d96e2-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:02.988247 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988181    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-trusted-ca\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.988568 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:02.988568 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988350    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-config\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.988568 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988364    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:02.988568 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988432    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb4c7102-dee6-436c-8d1a-3e4b83206308-serving-cert\") pod \"service-ca-operator-5d6c75f84f-45mdv\" (UID: \"cb4c7102-dee6-436c-8d1a-3e4b83206308\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:02.988568 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.988466    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sxjw\" (UniqueName: \"kubernetes.io/projected/f46daa6b-8762-4b6e-967f-2ae0d994f592-kube-api-access-8sxjw\") pod \"volume-data-source-validator-549c6d54cf-q4x7f\" (UID: \"f46daa6b-8762-4b6e-967f-2ae0d994f592\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f"
Jul 28 12:54:02.989088 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.989051    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:02.989474 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.989452    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-trusted-ca\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.990028 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.990011    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:02.991451 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.991431    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb4c7102-dee6-436c-8d1a-3e4b83206308-serving-cert\") pod \"service-ca-operator-5d6c75f84f-45mdv\" (UID: \"cb4c7102-dee6-436c-8d1a-3e4b83206308\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:02.991531 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.991452    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-serving-cert\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:02.992465 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.992448    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 12:54:02.992562 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.992517    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 12:54:02.992701 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.992687    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:54:02.992895 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.992879    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 12:54:02.992895 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.992888    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-k4vc7\""
Jul 28 12:54:02.998365 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:02.998344    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"]
Jul 28 12:54:03.004728 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.004702    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdgv\" (UniqueName: \"kubernetes.io/projected/b43f2f48-3db6-44ce-a879-e31a97f719d9-kube-api-access-rbdgv\") pod \"network-check-source-7d88cb55d8-gbqpx\" (UID: \"b43f2f48-3db6-44ce-a879-e31a97f719d9\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx"
Jul 28 12:54:03.004837 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.004770    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sxjw\" (UniqueName: \"kubernetes.io/projected/f46daa6b-8762-4b6e-967f-2ae0d994f592-kube-api-access-8sxjw\") pod \"volume-data-source-validator-549c6d54cf-q4x7f\" (UID: \"f46daa6b-8762-4b6e-967f-2ae0d994f592\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f"
Jul 28 12:54:03.006353 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.006337    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvkzf\" (UniqueName: \"kubernetes.io/projected/cb4c7102-dee6-436c-8d1a-3e4b83206308-kube-api-access-mvkzf\") pod \"service-ca-operator-5d6c75f84f-45mdv\" (UID: \"cb4c7102-dee6-436c-8d1a-3e4b83206308\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:03.006434 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.006372    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vzl\" (UniqueName: \"kubernetes.io/projected/289ee42c-3969-4bcb-90f5-c0fe0ea9562f-kube-api-access-g7vzl\") pod \"console-operator-584dcb6b54-zmt5l\" (UID: \"289ee42c-3969-4bcb-90f5-c0fe0ea9562f\") " pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:03.013646 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.013624    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"
Jul 28 12:54:03.033887 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.033867    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f"
Jul 28 12:54:03.058908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.058883    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:03.073246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.073212    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:03.083093 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.083055    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx"
Jul 28 12:54:03.089501 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089473    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-default-certificate\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.089632 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.089632 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089557    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx4st\" (UniqueName: \"kubernetes.io/projected/8f6ec1da-be3a-4780-9199-dc20768d96e2-kube-api-access-jx4st\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.089632 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzk8h\" (UniqueName: \"kubernetes.io/projected/4c4f466e-4360-449f-8ab8-da1732648943-kube-api-access-pzk8h\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:03.089794 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089637    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnw99\" (UniqueName: \"kubernetes.io/projected/0d851530-6f45-4c04-96c3-c95bbdf2241e-kube-api-access-cnw99\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.089794 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089661    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:03.089794 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089693    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6ec1da-be3a-4780-9199-dc20768d96e2-service-ca-bundle\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.089794 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089717    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6ec1da-be3a-4780-9199-dc20768d96e2-serving-cert\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.089794 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089758    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.090082 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089797    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6ec1da-be3a-4780-9199-dc20768d96e2-tmp\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.090082 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd4afe72-14f5-4172-b6e9-3eee63e1c5f6-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj\" (UID: \"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.090082 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089890    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f6ec1da-be3a-4780-9199-dc20768d96e2-snapshots\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.090082 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089914    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6ec1da-be3a-4780-9199-dc20768d96e2-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.090082 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.089988    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:03.090082 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.090040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd4afe72-14f5-4172-b6e9-3eee63e1c5f6-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj\" (UID: \"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.090082 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.090071    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkh5v\" (UniqueName: \"kubernetes.io/projected/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-kube-api-access-zkh5v\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:03.091459 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.090099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4c4f466e-4360-449f-8ab8-da1732648943-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:03.091459 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.090134    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxp7w\" (UniqueName: \"kubernetes.io/projected/fd4afe72-14f5-4172-b6e9-3eee63e1c5f6-kube-api-access-dxp7w\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj\" (UID: \"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.091459 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.090177    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-stats-auth\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.091459 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.090668    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6ec1da-be3a-4780-9199-dc20768d96e2-service-ca-bundle\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.091459 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.091135    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:54:03.091459 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.091201    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls podName:9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:03.591181032 +0000 UTC m=+42.914173951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls") pod "cluster-samples-operator-6748489656-4wxjt" (UID: "9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c") : secret "samples-operator-tls" not found
Jul 28 12:54:03.091900 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.091614    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6ec1da-be3a-4780-9199-dc20768d96e2-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.092131 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.092108    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:03.092200 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.092185    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:03.592151026 +0000 UTC m=+42.915143958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : secret "router-metrics-certs-default" not found
Jul 28 12:54:03.092364 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.092342    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:03.092438 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.092347    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6ec1da-be3a-4780-9199-dc20768d96e2-tmp\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.092438 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.092346    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:03.592329887 +0000 UTC m=+42.915322826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:03.092438 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.092438    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls podName:4c4f466e-4360-449f-8ab8-da1732648943 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:03.592423954 +0000 UTC m=+42.915416887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8kwgx" (UID: "4c4f466e-4360-449f-8ab8-da1732648943") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:03.092615 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.092460    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4c4f466e-4360-449f-8ab8-da1732648943-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:03.092764 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.092743    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f6ec1da-be3a-4780-9199-dc20768d96e2-snapshots\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.094338 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.094302    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-default-certificate\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.094600 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.094558    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-stats-auth\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.094796 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.094773    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6ec1da-be3a-4780-9199-dc20768d96e2-serving-cert\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.103319 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.103180    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzk8h\" (UniqueName: \"kubernetes.io/projected/4c4f466e-4360-449f-8ab8-da1732648943-kube-api-access-pzk8h\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:03.104371 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.104325    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnw99\" (UniqueName: \"kubernetes.io/projected/0d851530-6f45-4c04-96c3-c95bbdf2241e-kube-api-access-cnw99\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.105048 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.104728    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx4st\" (UniqueName: \"kubernetes.io/projected/8f6ec1da-be3a-4780-9199-dc20768d96e2-kube-api-access-jx4st\") pod \"insights-operator-5885fdb759-rgl5v\" (UID: \"8f6ec1da-be3a-4780-9199-dc20768d96e2\") " pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.106541 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.106494    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkh5v\" (UniqueName: \"kubernetes.io/projected/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-kube-api-access-zkh5v\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:03.151867 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.151831    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-rgl5v"
Jul 28 12:54:03.175747 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.175551    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:03.198080 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.192170    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv"]
Jul 28 12:54:03.198080 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.192799    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd4afe72-14f5-4172-b6e9-3eee63e1c5f6-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj\" (UID: \"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.198080 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.192849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxp7w\" (UniqueName: \"kubernetes.io/projected/fd4afe72-14f5-4172-b6e9-3eee63e1c5f6-kube-api-access-dxp7w\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj\" (UID: \"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.198080 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.193076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd4afe72-14f5-4172-b6e9-3eee63e1c5f6-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj\" (UID: \"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.198080 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.197405    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd4afe72-14f5-4172-b6e9-3eee63e1c5f6-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj\" (UID: \"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.205793 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.205752    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd4afe72-14f5-4172-b6e9-3eee63e1c5f6-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj\" (UID: \"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.208952 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.208908    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxp7w\" (UniqueName: \"kubernetes.io/projected/fd4afe72-14f5-4172-b6e9-3eee63e1c5f6-kube-api-access-dxp7w\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj\" (UID: \"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.209992 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:03.209964    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4c7102_dee6_436c_8d1a_3e4b83206308.slice/crio-4037611ba9458cc33afc9087cc0b45d7871e66a79a7f87a7dc039bd424c16971 WatchSource:0}: Error finding container 4037611ba9458cc33afc9087cc0b45d7871e66a79a7f87a7dc039bd424c16971: Status 404 returned error can't find the container with id 4037611ba9458cc33afc9087cc0b45d7871e66a79a7f87a7dc039bd424c16971
Jul 28 12:54:03.225932 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.225862    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f"]
Jul 28 12:54:03.255633 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.255563    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-zmt5l"]
Jul 28 12:54:03.260496 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.260368    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:03.260987 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:03.260963    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289ee42c_3969_4bcb_90f5_c0fe0ea9562f.slice/crio-79a0a050cd375dba25a4c56c7833393d37bb6e70571088a62281d6facd6bbee2 WatchSource:0}: Error finding container 79a0a050cd375dba25a4c56c7833393d37bb6e70571088a62281d6facd6bbee2: Status 404 returned error can't find the container with id 79a0a050cd375dba25a4c56c7833393d37bb6e70571088a62281d6facd6bbee2
Jul 28 12:54:03.286238 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.286210    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx"]
Jul 28 12:54:03.290156 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:03.290131    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43f2f48_3db6_44ce_a879_e31a97f719d9.slice/crio-fc81da98daa533dd3119fc2118057856c97893ace1a5cf6d52ecee6469c47512 WatchSource:0}: Error finding container fc81da98daa533dd3119fc2118057856c97893ace1a5cf6d52ecee6469c47512: Status 404 returned error can't find the container with id fc81da98daa533dd3119fc2118057856c97893ace1a5cf6d52ecee6469c47512
Jul 28 12:54:03.294507 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.294451    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:54:03.294629 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.294607    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:54:03.294629 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.294620    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:54:03.294822 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.294687    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls podName:7f5a807c-9e19-4693-acc3-20e97affcb82 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:11.294664043 +0000 UTC m=+50.617656962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls") pod "dns-default-5c2st" (UID: "7f5a807c-9e19-4693-acc3-20e97affcb82") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:03.294822 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.294721    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:54:03.294822 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.294731    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4bf4b95c-pzrf9: secret "image-registry-tls" not found
Jul 28 12:54:03.294822 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.294769    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls podName:0435f7c0-0018-49a2-b8c1-c4048884fc03 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:11.29475906 +0000 UTC m=+50.617751978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls") pod "image-registry-b4bf4b95c-pzrf9" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03") : secret "image-registry-tls" not found
Jul 28 12:54:03.304292 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.304267    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"
Jul 28 12:54:03.335881 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.335851    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rgl5v"]
Jul 28 12:54:03.338595 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:03.338570    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6ec1da_be3a_4780_9199_dc20768d96e2.slice/crio-26c86736ef743df82af70b1410a53329c1be642ed5667dfa3a75f9f57e581d88 WatchSource:0}: Error finding container 26c86736ef743df82af70b1410a53329c1be642ed5667dfa3a75f9f57e581d88: Status 404 returned error can't find the container with id 26c86736ef743df82af70b1410a53329c1be642ed5667dfa3a75f9f57e581d88
Jul 28 12:54:03.438980 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.438928    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx" event={"ID":"b43f2f48-3db6-44ce-a879-e31a97f719d9","Type":"ContainerStarted","Data":"fc81da98daa533dd3119fc2118057856c97893ace1a5cf6d52ecee6469c47512"}
Jul 28 12:54:03.441368 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.441338    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj"]
Jul 28 12:54:03.443075 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.443051    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mpcpt" event={"ID":"bd90f16b-a738-49f8-99f8-ac9981959493","Type":"ContainerStarted","Data":"6b80fca1b97f76f211bed2377b605d60e3966f2ecefd3b637c633c8098a7fc2b"}
Jul 28 12:54:03.444193 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.444172    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f" event={"ID":"f46daa6b-8762-4b6e-967f-2ae0d994f592","Type":"ContainerStarted","Data":"918f04a314cf5554a3616ebec475fed93750c2ee244c430b36cba9b90c56eb65"}
Jul 28 12:54:03.444494 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:03.444464    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4afe72_14f5_4172_b6e9_3eee63e1c5f6.slice/crio-446a5a1bfc501a4209e8e726699ec29557264a93ab312f0481fe04fba3f996b9 WatchSource:0}: Error finding container 446a5a1bfc501a4209e8e726699ec29557264a93ab312f0481fe04fba3f996b9: Status 404 returned error can't find the container with id 446a5a1bfc501a4209e8e726699ec29557264a93ab312f0481fe04fba3f996b9
Jul 28 12:54:03.445285 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.445262    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rgl5v" event={"ID":"8f6ec1da-be3a-4780-9199-dc20768d96e2","Type":"ContainerStarted","Data":"26c86736ef743df82af70b1410a53329c1be642ed5667dfa3a75f9f57e581d88"}
Jul 28 12:54:03.446300 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.446275    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" event={"ID":"289ee42c-3969-4bcb-90f5-c0fe0ea9562f","Type":"ContainerStarted","Data":"79a0a050cd375dba25a4c56c7833393d37bb6e70571088a62281d6facd6bbee2"}
Jul 28 12:54:03.447369 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.447349    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv" event={"ID":"cb4c7102-dee6-436c-8d1a-3e4b83206308","Type":"ContainerStarted","Data":"4037611ba9458cc33afc9087cc0b45d7871e66a79a7f87a7dc039bd424c16971"}
Jul 28 12:54:03.468734 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.468691    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mpcpt" podStartSLOduration=7.741144471 podStartE2EDuration="42.468674299s" totalImagesPullingTime="34.727529828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:03.468243033 +0000 UTC m=+42.791235974" watchObservedRunningTime="2026-07-28 12:54:03.468674299 +0000 UTC m=+42.791667240"
Jul 28 12:54:03.470374 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.470348    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:03.496772 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.496698    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:03.496916 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.496897    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:03.497009 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.496995    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert podName:25ff57f5-17a0-46b8-9fa6-6e6a05de1ded nodeName:}" failed. No retries permitted until 2026-07-28 12:54:04.496974772 +0000 UTC m=+43.819967700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9dkd5" (UID: "25ff57f5-17a0-46b8-9fa6-6e6a05de1ded") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:03.570018 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.569980    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:03.597232 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.597209    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:03.597360 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.597348    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:54:03.597410 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.597400    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls podName:9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:04.597385856 +0000 UTC m=+43.920378777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls") pod "cluster-samples-operator-6748489656-4wxjt" (UID: "9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c") : secret "samples-operator-tls" not found
Jul 28 12:54:03.597484 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.597457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.597571 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.597542    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:03.597626 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:03.597596    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:03.597671 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.597642    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:04.597617283 +0000 UTC m=+43.920610216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:03.597708 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.597680    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:03.597753 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.597688    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:03.597808 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.597734    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:04.597720446 +0000 UTC m=+43.920713365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : secret "router-metrics-certs-default" not found
Jul 28 12:54:03.597858 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:03.597830    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls podName:4c4f466e-4360-449f-8ab8-da1732648943 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:04.597808828 +0000 UTC m=+43.920801761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8kwgx" (UID: "4c4f466e-4360-449f-8ab8-da1732648943") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:04.264755 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.264718    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:04.365674 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.365573    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:04.452386 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.452294    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj" event={"ID":"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6","Type":"ContainerStarted","Data":"446a5a1bfc501a4209e8e726699ec29557264a93ab312f0481fe04fba3f996b9"}
Jul 28 12:54:04.506652 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.506613    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:04.507331 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:04.506878    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:04.507331 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:04.506959    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert podName:25ff57f5-17a0-46b8-9fa6-6e6a05de1ded nodeName:}" failed. No retries permitted until 2026-07-28 12:54:06.506925498 +0000 UTC m=+45.829918437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9dkd5" (UID: "25ff57f5-17a0-46b8-9fa6-6e6a05de1ded") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.608305    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.608401    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.608450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.608573    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:04.608702    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:04.608721    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:04.608760    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls podName:9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:06.608742691 +0000 UTC m=+45.931735614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls") pod "cluster-samples-operator-6748489656-4wxjt" (UID: "9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c") : secret "samples-operator-tls" not found
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:04.608777    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls podName:4c4f466e-4360-449f-8ab8-da1732648943 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:06.60876791 +0000 UTC m=+45.931760832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8kwgx" (UID: "4c4f466e-4360-449f-8ab8-da1732648943") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:04.608837    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:04.609297 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:04.608868    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:06.608857533 +0000 UTC m=+45.931850456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : secret "router-metrics-certs-default" not found
Jul 28 12:54:04.609956 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:04.609530    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:06.609512459 +0000 UTC m=+45.932505379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:04.676298 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.676259    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:04.763274 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:04.763234    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:05.074849 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:05.074813    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:06.526849 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:06.526812    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:06.527271 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:06.526979    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:06.527271 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:06.527051    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert podName:25ff57f5-17a0-46b8-9fa6-6e6a05de1ded nodeName:}" failed. No retries permitted until 2026-07-28 12:54:10.527036311 +0000 UTC m=+49.850029232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9dkd5" (UID: "25ff57f5-17a0-46b8-9fa6-6e6a05de1ded") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:06.627770 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:06.627736    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:06.627987 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:06.627795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:06.627987 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:06.627843    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:06.627987 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:06.627909    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:10.627892488 +0000 UTC m=+49.950885406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:06.627987 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:06.627968    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:06.628198 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:06.627972    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:06.628198 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:06.628001    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:06.628198 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:06.628025    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:10.628017479 +0000 UTC m=+49.951010398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : secret "router-metrics-certs-default" not found
Jul 28 12:54:06.628198 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:06.628050    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls podName:4c4f466e-4360-449f-8ab8-da1732648943 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:10.628035105 +0000 UTC m=+49.951028037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8kwgx" (UID: "4c4f466e-4360-449f-8ab8-da1732648943") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:06.628198 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:06.628050    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:54:06.628198 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:06.628097    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls podName:9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:10.628086575 +0000 UTC m=+49.951079496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls") pod "cluster-samples-operator-6748489656-4wxjt" (UID: "9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c") : secret "samples-operator-tls" not found
Jul 28 12:54:10.471758 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.471685    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rgl5v" event={"ID":"8f6ec1da-be3a-4780-9199-dc20768d96e2","Type":"ContainerStarted","Data":"040a40f1418ec8894ae8d042847f453479f44b04e34894b9935377417d03c5d8"}
Jul 28 12:54:10.473548 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.473528    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/0.log"
Jul 28 12:54:10.473656 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.473563    2572 generic.go:356] "Generic (PLEG): container finished" podID="289ee42c-3969-4bcb-90f5-c0fe0ea9562f" containerID="943535b0961b565fb8342d028647aaa9eaa2912250cd349c9ea59b04bf00c485" exitCode=255
Jul 28 12:54:10.473770 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.473748    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" event={"ID":"289ee42c-3969-4bcb-90f5-c0fe0ea9562f","Type":"ContainerDied","Data":"943535b0961b565fb8342d028647aaa9eaa2912250cd349c9ea59b04bf00c485"}
Jul 28 12:54:10.473865 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.473852    2572 scope.go:117] "RemoveContainer" containerID="943535b0961b565fb8342d028647aaa9eaa2912250cd349c9ea59b04bf00c485"
Jul 28 12:54:10.475383 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.475357    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj" event={"ID":"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6","Type":"ContainerStarted","Data":"fab670efa39560444efb173ea6cfd6c73eb5429e6aba7e0c15919877e9bee02c"}
Jul 28 12:54:10.477298 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.477274    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv" event={"ID":"cb4c7102-dee6-436c-8d1a-3e4b83206308","Type":"ContainerStarted","Data":"f8d7d4a745358d153906f5d2213391bc1c72673146490b00ebcf30b65187f3c9"}
Jul 28 12:54:10.479086 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.479056    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx" event={"ID":"b43f2f48-3db6-44ce-a879-e31a97f719d9","Type":"ContainerStarted","Data":"3b9891e2cc96002f0199660ff6ad6c9ad4b39d0402b3af433a16be81ef85eb99"}
Jul 28 12:54:10.480610 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.480586    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f" event={"ID":"f46daa6b-8762-4b6e-967f-2ae0d994f592","Type":"ContainerStarted","Data":"f2507e8d2c1cef59636b8f3a8ea6b95395e48c559d7ceadfde19b0128aa72403"}
Jul 28 12:54:10.497357 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.497313    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-rgl5v" podStartSLOduration=1.682521494 podStartE2EDuration="8.497294406s" totalImagesPullingTime="6.814772912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:10.496480794 +0000 UTC m=+49.819473735" watchObservedRunningTime="2026-07-28 12:54:10.497294406 +0000 UTC m=+49.820287328"
Jul 28 12:54:10.522235 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.520885    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gbqpx" podStartSLOduration=1.6558232259999999 podStartE2EDuration="8.520863981s" totalImagesPullingTime="6.865040755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:10.520283899 +0000 UTC m=+49.843276841" watchObservedRunningTime="2026-07-28 12:54:10.520863981 +0000 UTC m=+49.843856902"
Jul 28 12:54:10.537234 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.537175    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj" podStartSLOduration=1.825559438 podStartE2EDuration="8.537155189s" totalImagesPullingTime="6.711595751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:10.536816194 +0000 UTC m=+49.859809134" watchObservedRunningTime="2026-07-28 12:54:10.537155189 +0000 UTC m=+49.860148130"
Jul 28 12:54:10.554272 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.554225    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv" podStartSLOduration=1.6095287489999999 podStartE2EDuration="8.554209818s" totalImagesPullingTime="6.944681069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:10.552973184 +0000 UTC m=+49.875966125" watchObservedRunningTime="2026-07-28 12:54:10.554209818 +0000 UTC m=+49.877202760"
Jul 28 12:54:10.567498 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.565236    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:10.567498 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:10.566032    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:10.567498 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:10.566095    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert podName:25ff57f5-17a0-46b8-9fa6-6e6a05de1ded nodeName:}" failed. No retries permitted until 2026-07-28 12:54:18.566075686 +0000 UTC m=+57.889068610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9dkd5" (UID: "25ff57f5-17a0-46b8-9fa6-6e6a05de1ded") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:10.567752 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.567692    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q4x7f" podStartSLOduration=1.6405856330000002 podStartE2EDuration="8.567672755s" totalImagesPullingTime="6.927087122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:10.567312325 +0000 UTC m=+49.890305266" watchObservedRunningTime="2026-07-28 12:54:10.567672755 +0000 UTC m=+49.890665700"
Jul 28 12:54:10.666780 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.666744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:10.666951 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.666797    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:10.666951 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.666873    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:10.666951 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:10.666909    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:10.667114 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.666955    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:10.667114 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:10.666989    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls podName:4c4f466e-4360-449f-8ab8-da1732648943 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:18.666968141 +0000 UTC m=+57.989961082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8kwgx" (UID: "4c4f466e-4360-449f-8ab8-da1732648943") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:10.667114 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:10.667040    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:18.667029649 +0000 UTC m=+57.990022569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:10.667114 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:10.667097    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:10.667322 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:10.667126    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:18.66711635 +0000 UTC m=+57.990109269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : secret "router-metrics-certs-default" not found
Jul 28 12:54:10.667322 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:10.667157    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:54:10.667322 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:10.667224    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls podName:9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:18.667207123 +0000 UTC m=+57.990200044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls") pod "cluster-samples-operator-6748489656-4wxjt" (UID: "9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c") : secret "samples-operator-tls" not found
Jul 28 12:54:10.728381 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.728329    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:10.829852 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:10.829810    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:11.373046 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.373005    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:54:11.373211 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.373065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:54:11.373211 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:11.373172    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:54:11.373211 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:11.373181    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:54:11.373211 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:11.373204    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4bf4b95c-pzrf9: secret "image-registry-tls" not found
Jul 28 12:54:11.373345 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:11.373225    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls podName:7f5a807c-9e19-4693-acc3-20e97affcb82 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:27.373211834 +0000 UTC m=+66.696204752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls") pod "dns-default-5c2st" (UID: "7f5a807c-9e19-4693-acc3-20e97affcb82") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:11.373345 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:11.373258    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls podName:0435f7c0-0018-49a2-b8c1-c4048884fc03 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:27.373240396 +0000 UTC m=+66.696233317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls") pod "image-registry-b4bf4b95c-pzrf9" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03") : secret "image-registry-tls" not found
Jul 28 12:54:11.485748 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.485671    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/1.log"
Jul 28 12:54:11.486193 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.486125    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/0.log"
Jul 28 12:54:11.486193 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.486163    2572 generic.go:356] "Generic (PLEG): container finished" podID="289ee42c-3969-4bcb-90f5-c0fe0ea9562f" containerID="f5cb523002147196a2973f14b32f778e96c5d56eaa37cb868b6a78542fd9ded6" exitCode=255
Jul 28 12:54:11.486296 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.486258    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" event={"ID":"289ee42c-3969-4bcb-90f5-c0fe0ea9562f","Type":"ContainerDied","Data":"f5cb523002147196a2973f14b32f778e96c5d56eaa37cb868b6a78542fd9ded6"}
Jul 28 12:54:11.486350 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.486335    2572 scope.go:117] "RemoveContainer" containerID="943535b0961b565fb8342d028647aaa9eaa2912250cd349c9ea59b04bf00c485"
Jul 28 12:54:11.486619 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.486595    2572 scope.go:117] "RemoveContainer" containerID="f5cb523002147196a2973f14b32f778e96c5d56eaa37cb868b6a78542fd9ded6"
Jul 28 12:54:11.486809 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:11.486784    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-zmt5l_openshift-console-operator(289ee42c-3969-4bcb-90f5-c0fe0ea9562f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" podUID="289ee42c-3969-4bcb-90f5-c0fe0ea9562f"
Jul 28 12:54:11.615167 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.615125    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:11.661211 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.661175    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr"]
Jul 28 12:54:11.680706 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.680678    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr"]
Jul 28 12:54:11.680868 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.680799    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr"
Jul 28 12:54:11.683558 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.683132    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-42vn2\""
Jul 28 12:54:11.683558 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.683396    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 12:54:11.683776 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.683757    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 12:54:11.777813 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.777778    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78tj\" (UniqueName: \"kubernetes.io/projected/3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab-kube-api-access-l78tj\") pod \"migrator-6865c9967-rb7cr\" (UID: \"3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr"
Jul 28 12:54:11.879241 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.879205    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78tj\" (UniqueName: \"kubernetes.io/projected/3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab-kube-api-access-l78tj\") pod \"migrator-6865c9967-rb7cr\" (UID: \"3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr"
Jul 28 12:54:11.887715 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.887689    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78tj\" (UniqueName: \"kubernetes.io/projected/3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab-kube-api-access-l78tj\") pod \"migrator-6865c9967-rb7cr\" (UID: \"3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr"
Jul 28 12:54:11.991143 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:11.991099    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr"
Jul 28 12:54:12.036231 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.036160    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lcbtm"]
Jul 28 12:54:12.060805 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.060012    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lcbtm"]
Jul 28 12:54:12.060805 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.060158    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.062734 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.062705    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:54:12.062861 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.062711    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-blfqk\""
Jul 28 12:54:12.063573 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.063550    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:54:12.082523 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.082097    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.082523 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.082299    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngtgl\" (UniqueName: \"kubernetes.io/projected/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-api-access-ngtgl\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.082523 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.082365    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.082523 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.082457    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08ef64bf-f32f-4215-8259-6a70cf9f3f84-data-volume\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.082822 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.082580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08ef64bf-f32f-4215-8259-6a70cf9f3f84-crio-socket\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.156994 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.156969    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr"]
Jul 28 12:54:12.160890 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:12.160856    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae1eed1_71ad_4bb2_9b98_7c0323b2e7ab.slice/crio-b0e80a76cf70a10b2887df2d46b75e68c9a4488f7784fb56283e5552ccd70518 WatchSource:0}: Error finding container b0e80a76cf70a10b2887df2d46b75e68c9a4488f7784fb56283e5552ccd70518: Status 404 returned error can't find the container with id b0e80a76cf70a10b2887df2d46b75e68c9a4488f7784fb56283e5552ccd70518
Jul 28 12:54:12.183381 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.183351    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.183539 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:12.183507    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:12.183539 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.183530    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngtgl\" (UniqueName: \"kubernetes.io/projected/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-api-access-ngtgl\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.183652 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.183572    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.183652 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:12.183617    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:12.683596225 +0000 UTC m=+52.006589149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:12.184031 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:12.184006    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:12.184174 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.184084    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08ef64bf-f32f-4215-8259-6a70cf9f3f84-data-volume\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.184238 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:12.184209    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:12.684166218 +0000 UTC m=+52.007159145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:12.184304 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.184252    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08ef64bf-f32f-4215-8259-6a70cf9f3f84-crio-socket\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.184391 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.184372    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08ef64bf-f32f-4215-8259-6a70cf9f3f84-crio-socket\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.184391 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.184385    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08ef64bf-f32f-4215-8259-6a70cf9f3f84-data-volume\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.193448 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.193419    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngtgl\" (UniqueName: \"kubernetes.io/projected/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-api-access-ngtgl\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.490624 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.490543    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr" event={"ID":"3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab","Type":"ContainerStarted","Data":"b0e80a76cf70a10b2887df2d46b75e68c9a4488f7784fb56283e5552ccd70518"}
Jul 28 12:54:12.491895 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.491877    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/1.log"
Jul 28 12:54:12.492221 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.492205    2572 scope.go:117] "RemoveContainer" containerID="f5cb523002147196a2973f14b32f778e96c5d56eaa37cb868b6a78542fd9ded6"
Jul 28 12:54:12.492373 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:12.492355    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-zmt5l_openshift-console-operator(289ee42c-3969-4bcb-90f5-c0fe0ea9562f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" podUID="289ee42c-3969-4bcb-90f5-c0fe0ea9562f"
Jul 28 12:54:12.689212 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.689165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.689398 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:12.689295    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:12.689398 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:12.689298    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:12.689520 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:12.689446    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:13.689426302 +0000 UTC m=+53.012419242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:12.689520 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:12.689333    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:12.689520 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:12.689514    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:13.689500711 +0000 UTC m=+53.012493629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:13.059218 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:13.059177    2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:13.059403 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:13.059251    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:13.496282 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:13.496202    2572 scope.go:117] "RemoveContainer" containerID="f5cb523002147196a2973f14b32f778e96c5d56eaa37cb868b6a78542fd9ded6"
Jul 28 12:54:13.496714 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:13.496400    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-zmt5l_openshift-console-operator(289ee42c-3969-4bcb-90f5-c0fe0ea9562f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" podUID="289ee42c-3969-4bcb-90f5-c0fe0ea9562f"
Jul 28 12:54:13.699322 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:13.699295    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:13.699450 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:13.699392    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:13.699514 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:13.699474    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:13.699514 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:13.699491    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:13.699614 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:13.699567    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:15.699544579 +0000 UTC m=+55.022537504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:13.699614 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:13.699585    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:15.699576559 +0000 UTC m=+55.022569476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:14.029738 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:14.029708    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-szhdp_ad991f88-1a5a-4745-b3c6-7f00b9f4ee96/dns-node-resolver/0.log"
Jul 28 12:54:14.501183 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:14.501097    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr" event={"ID":"3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab","Type":"ContainerStarted","Data":"a5b56ad9b51c2dd93fba235c3b35bf82088a99a8f1547ba2a90d025ba5c21c25"}
Jul 28 12:54:14.501183 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:14.501150    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr" event={"ID":"3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab","Type":"ContainerStarted","Data":"52f95990ab09c68f5339d74f1dc6e9dab7d0a1d6a62132b16a4590943e61d7d4"}
Jul 28 12:54:14.520914 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:14.520870    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rb7cr" podStartSLOduration=2.078212978 podStartE2EDuration="3.520856871s" totalImagesPullingTime="1.442643893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:14.519986046 +0000 UTC m=+53.842978985" watchObservedRunningTime="2026-07-28 12:54:14.520856871 +0000 UTC m=+53.843849811"
Jul 28 12:54:15.029001 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:15.028976    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ksgqt_c0e02d46-2f01-4d45-b048-f0dc22cfad61/node-ca/0.log"
Jul 28 12:54:15.629746 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:15.629719    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5s4dp_c1ea0458-166f-4cab-a151-ae16ad59c313/serve-healthcheck-canary/0.log"
Jul 28 12:54:15.717473 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:15.717436    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:15.717622 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:15.717518    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:15.717622 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:15.717582    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:15.717622 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:15.717607    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:15.717794 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:15.717644    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:19.717629482 +0000 UTC m=+59.040622400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:15.717794 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:15.717662    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:19.717654051 +0000 UTC m=+59.040646969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:15.830773 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:15.830739    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-rb7cr_3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab/migrator/0.log"
Jul 28 12:54:16.031127 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:16.031100    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-rb7cr_3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab/graceful-termination/0.log"
Jul 28 12:54:16.235651 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:16.235624    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj_fd4afe72-14f5-4172-b6e9-3eee63e1c5f6/kube-storage-version-migrator-operator/0.log"
Jul 28 12:54:18.642534 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:18.642478    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:18.643048 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:18.642632    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:18.643048 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:18.642711    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert podName:25ff57f5-17a0-46b8-9fa6-6e6a05de1ded nodeName:}" failed. No retries permitted until 2026-07-28 12:54:34.642696537 +0000 UTC m=+73.965689455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-9dkd5" (UID: "25ff57f5-17a0-46b8-9fa6-6e6a05de1ded") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:18.743279 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:18.743231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:18.743402 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:18.743326    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:18.743402 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:18.743363    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:18.743491 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:18.743396    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:34.743378006 +0000 UTC m=+74.066370928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:18.743491 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:18.743457    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:18.743491 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:18.743460    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:18.743604 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:18.743510    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:18.743604 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:18.743535    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 12:54:18.743604 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:18.743562    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls podName:9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:34.743550648 +0000 UTC m=+74.066543566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls") pod "cluster-samples-operator-6748489656-4wxjt" (UID: "9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c") : secret "samples-operator-tls" not found
Jul 28 12:54:18.743604 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:18.743580    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs podName:0d851530-6f45-4c04-96c3-c95bbdf2241e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:34.743567413 +0000 UTC m=+74.066560330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs") pod "router-default-8b6d76f97-jh5t9" (UID: "0d851530-6f45-4c04-96c3-c95bbdf2241e") : secret "router-metrics-certs-default" not found
Jul 28 12:54:18.743740 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:18.743607    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls podName:4c4f466e-4360-449f-8ab8-da1732648943 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:34.743588068 +0000 UTC m=+74.066580986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8kwgx" (UID: "4c4f466e-4360-449f-8ab8-da1732648943") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:19.752710 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:19.752651    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:19.753140 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:19.752774    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:19.753140 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:19.752795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:19.753140 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:19.752863    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:27.752848337 +0000 UTC m=+67.075841258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:19.753140 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:19.752972    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:19.753140 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:19.753032    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:27.753015675 +0000 UTC m=+67.076008619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:22.409535 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:22.409505    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4krx"
Jul 28 12:54:25.908612 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:25.908564    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:54:25.909053 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:25.908646    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:54:25.909053 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:25.908682    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:54:25.911095 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:25.911066    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt77l\" (UniqueName: \"kubernetes.io/projected/64cdd13d-520d-4f12-86fa-a5f99d362fba-kube-api-access-rt77l\") pod \"network-check-target-lkksh\" (UID: \"64cdd13d-520d-4f12-86fa-a5f99d362fba\") " pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:54:25.911413 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:25.911393    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:54:25.911511 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:25.911470    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:54:25.919184 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:25.919168    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 12:54:25.919272 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:25.919255    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs podName:43fa5143-75d6-4f79-88d0-97b5a5ecf09c nodeName:}" failed. No retries permitted until 2026-07-28 12:55:29.919234599 +0000 UTC m=+129.242227529 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs") pod "network-metrics-daemon-k4prs" (UID: "43fa5143-75d6-4f79-88d0-97b5a5ecf09c") : secret "metrics-daemon-secret" not found
Jul 28 12:54:25.921232 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:25.921214    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/64386eb9-8899-410e-adf0-3f21a2a226b0-original-pull-secret\") pod \"global-pull-secret-syncer-p6smg\" (UID: \"64386eb9-8899-410e-adf0-3f21a2a226b0\") " pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:54:25.989084 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:25.989045    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p6smg"
Jul 28 12:54:26.003645 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:26.003620    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kmztt\""
Jul 28 12:54:26.011706 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:26.011675    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:54:26.125081 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:26.125026    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p6smg"]
Jul 28 12:54:26.128848 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:26.128822    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64386eb9_8899_410e_adf0_3f21a2a226b0.slice/crio-4de4275e7d95f659c1ff09bd05ba7e6d3bbf83bcdbd6b53d5bf8b15d12e84149 WatchSource:0}: Error finding container 4de4275e7d95f659c1ff09bd05ba7e6d3bbf83bcdbd6b53d5bf8b15d12e84149: Status 404 returned error can't find the container with id 4de4275e7d95f659c1ff09bd05ba7e6d3bbf83bcdbd6b53d5bf8b15d12e84149
Jul 28 12:54:26.140988 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:26.140962    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lkksh"]
Jul 28 12:54:26.144015 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:26.143991    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cdd13d_520d_4f12_86fa_a5f99d362fba.slice/crio-0ac249ffe8d1354596310e23d73e75df66fb3d3a0a870c310766df1b76691ed9 WatchSource:0}: Error finding container 0ac249ffe8d1354596310e23d73e75df66fb3d3a0a870c310766df1b76691ed9: Status 404 returned error can't find the container with id 0ac249ffe8d1354596310e23d73e75df66fb3d3a0a870c310766df1b76691ed9
Jul 28 12:54:26.541953 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:26.541899    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lkksh" event={"ID":"64cdd13d-520d-4f12-86fa-a5f99d362fba","Type":"ContainerStarted","Data":"006b42bdbaed01d411537ee1d06772c57bee66fd16458090529356f2a3ee5523"}
Jul 28 12:54:26.542134 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:26.541984    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lkksh" event={"ID":"64cdd13d-520d-4f12-86fa-a5f99d362fba","Type":"ContainerStarted","Data":"0ac249ffe8d1354596310e23d73e75df66fb3d3a0a870c310766df1b76691ed9"}
Jul 28 12:54:26.542134 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:26.542011    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:54:26.543290 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:26.543256    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p6smg" event={"ID":"64386eb9-8899-410e-adf0-3f21a2a226b0","Type":"ContainerStarted","Data":"4de4275e7d95f659c1ff09bd05ba7e6d3bbf83bcdbd6b53d5bf8b15d12e84149"}
Jul 28 12:54:26.558240 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:26.558182    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lkksh" podStartSLOduration=65.558162151 podStartE2EDuration="1m5.558162151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:26.55772857 +0000 UTC m=+65.880721512" watchObservedRunningTime="2026-07-28 12:54:26.558162151 +0000 UTC m=+65.881155093"
Jul 28 12:54:27.175154 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.175125    2572 scope.go:117] "RemoveContainer" containerID="f5cb523002147196a2973f14b32f778e96c5d56eaa37cb868b6a78542fd9ded6"
Jul 28 12:54:27.425246 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.425160    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:54:27.425396 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.425334    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:54:27.428490 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.428217    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f5a807c-9e19-4693-acc3-20e97affcb82-metrics-tls\") pod \"dns-default-5c2st\" (UID: \"7f5a807c-9e19-4693-acc3-20e97affcb82\") " pod="openshift-dns/dns-default-5c2st"
Jul 28 12:54:27.428490 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.428328    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"image-registry-b4bf4b95c-pzrf9\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") " pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:54:27.547322 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.547294    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 12:54:27.547697 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.547676    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/1.log"
Jul 28 12:54:27.547762 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.547715    2572 generic.go:356] "Generic (PLEG): container finished" podID="289ee42c-3969-4bcb-90f5-c0fe0ea9562f" containerID="5159f62f34c2695f827f66422fcd7acd5d886bb05b0097a684781b2c19502609" exitCode=255
Jul 28 12:54:27.547819 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.547796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" event={"ID":"289ee42c-3969-4bcb-90f5-c0fe0ea9562f","Type":"ContainerDied","Data":"5159f62f34c2695f827f66422fcd7acd5d886bb05b0097a684781b2c19502609"}
Jul 28 12:54:27.547872 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.547859    2572 scope.go:117] "RemoveContainer" containerID="f5cb523002147196a2973f14b32f778e96c5d56eaa37cb868b6a78542fd9ded6"
Jul 28 12:54:27.548151 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.548133    2572 scope.go:117] "RemoveContainer" containerID="5159f62f34c2695f827f66422fcd7acd5d886bb05b0097a684781b2c19502609"
Jul 28 12:54:27.548349 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:27.548320    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-zmt5l_openshift-console-operator(289ee42c-3969-4bcb-90f5-c0fe0ea9562f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" podUID="289ee42c-3969-4bcb-90f5-c0fe0ea9562f"
Jul 28 12:54:27.622291 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.622258    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fzfhc\""
Jul 28 12:54:27.631076 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.631055    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:54:27.648556 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.648523    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xjd7z\""
Jul 28 12:54:27.657330 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.657295    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c2st"
Jul 28 12:54:27.808775 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.808663    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4bf4b95c-pzrf9"]
Jul 28 12:54:27.812020 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:27.811988    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0435f7c0_0018_49a2_b8c1_c4048884fc03.slice/crio-1df855f8c173dd8c304921448c54fc18cb232dacdfba0de6e9e01cc5917f6831 WatchSource:0}: Error finding container 1df855f8c173dd8c304921448c54fc18cb232dacdfba0de6e9e01cc5917f6831: Status 404 returned error can't find the container with id 1df855f8c173dd8c304921448c54fc18cb232dacdfba0de6e9e01cc5917f6831
Jul 28 12:54:27.836010 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.835980    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:27.836150 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.836131    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:27.836307 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:27.836286    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:27.836412 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:27.836354    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls podName:08ef64bf-f32f-4215-8259-6a70cf9f3f84 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:43.836333853 +0000 UTC m=+83.159326771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcbtm" (UID: "08ef64bf-f32f-4215-8259-6a70cf9f3f84") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:27.837048 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.837003    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08ef64bf-f32f-4215-8259-6a70cf9f3f84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:27.837935 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:27.837912    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c2st"]
Jul 28 12:54:27.846094 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:27.846066    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5a807c_9e19_4693_acc3_20e97affcb82.slice/crio-e7fe2c9ee889db3174a377483455ceac0217da4df943523cebd4a321a2d79086 WatchSource:0}: Error finding container e7fe2c9ee889db3174a377483455ceac0217da4df943523cebd4a321a2d79086: Status 404 returned error can't find the container with id e7fe2c9ee889db3174a377483455ceac0217da4df943523cebd4a321a2d79086
Jul 28 12:54:28.560039 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:28.559999    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 12:54:28.562136 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:28.562101    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c2st" event={"ID":"7f5a807c-9e19-4693-acc3-20e97affcb82","Type":"ContainerStarted","Data":"e7fe2c9ee889db3174a377483455ceac0217da4df943523cebd4a321a2d79086"}
Jul 28 12:54:28.563884 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:28.563845    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9" event={"ID":"0435f7c0-0018-49a2-b8c1-c4048884fc03","Type":"ContainerStarted","Data":"d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866"}
Jul 28 12:54:28.564007 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:28.563897    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9" event={"ID":"0435f7c0-0018-49a2-b8c1-c4048884fc03","Type":"ContainerStarted","Data":"1df855f8c173dd8c304921448c54fc18cb232dacdfba0de6e9e01cc5917f6831"}
Jul 28 12:54:28.564172 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:28.564152    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:54:28.586854 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:28.586802    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9" podStartSLOduration=67.586784156 podStartE2EDuration="1m7.586784156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:28.586303109 +0000 UTC m=+67.909296052" watchObservedRunningTime="2026-07-28 12:54:28.586784156 +0000 UTC m=+67.909777095"
Jul 28 12:54:30.574261 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:30.574229    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p6smg" event={"ID":"64386eb9-8899-410e-adf0-3f21a2a226b0","Type":"ContainerStarted","Data":"3b7ea2f713b8192508a2b640eeadd53dee50fccbc60b20eb6d1c474734c4cb84"}
Jul 28 12:54:30.576057 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:30.576031    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c2st" event={"ID":"7f5a807c-9e19-4693-acc3-20e97affcb82","Type":"ContainerStarted","Data":"c490ee191ac352ae3b368047ba61c664e982e23d5c8f3ad039177b4423ab94e8"}
Jul 28 12:54:30.576192 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:30.576079    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c2st" event={"ID":"7f5a807c-9e19-4693-acc3-20e97affcb82","Type":"ContainerStarted","Data":"7bd185fdf2c5c573c70cbe7b7e4cd7d00666a3f2c8949827ab1bdbffd30a37c9"}
Jul 28 12:54:30.576192 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:30.576161    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5c2st"
Jul 28 12:54:30.605474 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:30.605251    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p6smg" podStartSLOduration=65.382759992 podStartE2EDuration="1m9.605230439s" totalImagesPullingTime="4.222470447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:30.589568401 +0000 UTC m=+69.912561342" watchObservedRunningTime="2026-07-28 12:54:30.605230439 +0000 UTC m=+69.928223379"
Jul 28 12:54:30.605840 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:30.605808    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5c2st" podStartSLOduration=33.106633861 podStartE2EDuration="35.605799915s" totalImagesPullingTime="2.499166054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:30.604021981 +0000 UTC m=+69.927014921" watchObservedRunningTime="2026-07-28 12:54:30.605799915 +0000 UTC m=+69.928792855"
Jul 28 12:54:33.060057 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:33.060019    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:33.060476 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:33.060132    2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:33.060528 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:33.060481    2572 scope.go:117] "RemoveContainer" containerID="5159f62f34c2695f827f66422fcd7acd5d886bb05b0097a684781b2c19502609"
Jul 28 12:54:33.060658 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:33.060632    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-zmt5l_openshift-console-operator(289ee42c-3969-4bcb-90f5-c0fe0ea9562f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" podUID="289ee42c-3969-4bcb-90f5-c0fe0ea9562f"
Jul 28 12:54:34.697696 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.697657    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:34.700177 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.700151    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ff57f5-17a0-46b8-9fa6-6e6a05de1ded-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-9dkd5\" (UID: \"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:34.798758 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.798717    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:34.798915 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.798776    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:34.798915 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.798820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:34.798915 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.798879    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:34.799508 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.799480    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d851530-6f45-4c04-96c3-c95bbdf2241e-service-ca-bundle\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:34.801284 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.801256    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c4f466e-4360-449f-8ab8-da1732648943-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8kwgx\" (UID: \"4c4f466e-4360-449f-8ab8-da1732648943\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:34.801380 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.801301    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-4wxjt\" (UID: \"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:34.801380 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.801370    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d851530-6f45-4c04-96c3-c95bbdf2241e-metrics-certs\") pod \"router-default-8b6d76f97-jh5t9\" (UID: \"0d851530-6f45-4c04-96c3-c95bbdf2241e\") " pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:34.927000 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.926964    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vlcxf\""
Jul 28 12:54:34.935234 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.935208    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"
Jul 28 12:54:34.935539 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.935522    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kg6wq\""
Jul 28 12:54:34.943591 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.943564    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"
Jul 28 12:54:34.972772 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.972751    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c2ppd\""
Jul 28 12:54:34.980898 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:34.980872    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:35.010536 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.010510    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5zjx6\""
Jul 28 12:54:35.018792 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.018402    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"
Jul 28 12:54:35.121203 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.121124    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5"]
Jul 28 12:54:35.125477 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:35.125445    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ff57f5_17a0_46b8_9fa6_6e6a05de1ded.slice/crio-9648b85f108a50e0926936a827bdfc77028255c7012f6e43cc2735595302cf11 WatchSource:0}: Error finding container 9648b85f108a50e0926936a827bdfc77028255c7012f6e43cc2735595302cf11: Status 404 returned error can't find the container with id 9648b85f108a50e0926936a827bdfc77028255c7012f6e43cc2735595302cf11
Jul 28 12:54:35.146100 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.145996    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt"]
Jul 28 12:54:35.171423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.171390    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8b6d76f97-jh5t9"]
Jul 28 12:54:35.176207 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:35.176181    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d851530_6f45_4c04_96c3_c95bbdf2241e.slice/crio-db8b4b56b8949a13ad2a2d1a909b6b60be6d161f4c1375ebcc31380c6fb2d35f WatchSource:0}: Error finding container db8b4b56b8949a13ad2a2d1a909b6b60be6d161f4c1375ebcc31380c6fb2d35f: Status 404 returned error can't find the container with id db8b4b56b8949a13ad2a2d1a909b6b60be6d161f4c1375ebcc31380c6fb2d35f
Jul 28 12:54:35.197283 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.197252    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx"]
Jul 28 12:54:35.200346 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:35.200323    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4f466e_4360_449f_8ab8_da1732648943.slice/crio-369894f8e23355979462ae86262576b812ae6c210aa9af0e65363dcf5bfc2d93 WatchSource:0}: Error finding container 369894f8e23355979462ae86262576b812ae6c210aa9af0e65363dcf5bfc2d93: Status 404 returned error can't find the container with id 369894f8e23355979462ae86262576b812ae6c210aa9af0e65363dcf5bfc2d93
Jul 28 12:54:35.593597 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.593561    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt" event={"ID":"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c","Type":"ContainerStarted","Data":"34a6f41ac07ea8b5a371c403068df8e8f46f4b815b9645efba86a7584f5394f7"}
Jul 28 12:54:35.594727 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.594695    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5" event={"ID":"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded","Type":"ContainerStarted","Data":"9648b85f108a50e0926936a827bdfc77028255c7012f6e43cc2735595302cf11"}
Jul 28 12:54:35.595755 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.595738    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx" event={"ID":"4c4f466e-4360-449f-8ab8-da1732648943","Type":"ContainerStarted","Data":"369894f8e23355979462ae86262576b812ae6c210aa9af0e65363dcf5bfc2d93"}
Jul 28 12:54:35.597058 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.597039    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8b6d76f97-jh5t9" event={"ID":"0d851530-6f45-4c04-96c3-c95bbdf2241e","Type":"ContainerStarted","Data":"8616cd55f454550d8d0a193d1f16e6de0e7b774e3efcfd818d302adc83ebde73"}
Jul 28 12:54:35.597149 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.597069    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8b6d76f97-jh5t9" event={"ID":"0d851530-6f45-4c04-96c3-c95bbdf2241e","Type":"ContainerStarted","Data":"db8b4b56b8949a13ad2a2d1a909b6b60be6d161f4c1375ebcc31380c6fb2d35f"}
Jul 28 12:54:35.616636 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.616590    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8b6d76f97-jh5t9" podStartSLOduration=33.616575329 podStartE2EDuration="33.616575329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:35.615059691 +0000 UTC m=+74.938052631" watchObservedRunningTime="2026-07-28 12:54:35.616575329 +0000 UTC m=+74.939568269"
Jul 28 12:54:35.982052 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.981963    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:35.984541 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:35.984515    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:36.602924 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:36.602892    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:36.604200 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:36.604175    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8b6d76f97-jh5t9"
Jul 28 12:54:37.594588 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.594553    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"]
Jul 28 12:54:37.612092 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.612064    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"
Jul 28 12:54:37.615662 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.615423    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 12:54:37.615662 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.615460    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 12:54:37.615662 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.615546    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 12:54:37.615662 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.615434    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 12:54:37.615662 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.615646    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-k9qbd\""
Jul 28 12:54:37.615973 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.615868    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"]
Jul 28 12:54:37.725633 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.725594    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cs2d\" (UniqueName: \"kubernetes.io/projected/2dd9546e-8cc6-4066-8aa2-7640b678d59a-kube-api-access-4cs2d\") pod \"managed-serviceaccount-addon-agent-5858659ffd-ds9xt\" (UID: \"2dd9546e-8cc6-4066-8aa2-7640b678d59a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"
Jul 28 12:54:37.725812 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.725686    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dd9546e-8cc6-4066-8aa2-7640b678d59a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5858659ffd-ds9xt\" (UID: \"2dd9546e-8cc6-4066-8aa2-7640b678d59a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"
Jul 28 12:54:37.826309 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.826264    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cs2d\" (UniqueName: \"kubernetes.io/projected/2dd9546e-8cc6-4066-8aa2-7640b678d59a-kube-api-access-4cs2d\") pod \"managed-serviceaccount-addon-agent-5858659ffd-ds9xt\" (UID: \"2dd9546e-8cc6-4066-8aa2-7640b678d59a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"
Jul 28 12:54:37.826499 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.826358    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dd9546e-8cc6-4066-8aa2-7640b678d59a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5858659ffd-ds9xt\" (UID: \"2dd9546e-8cc6-4066-8aa2-7640b678d59a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"
Jul 28 12:54:37.829404 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.829374    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dd9546e-8cc6-4066-8aa2-7640b678d59a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5858659ffd-ds9xt\" (UID: \"2dd9546e-8cc6-4066-8aa2-7640b678d59a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"
Jul 28 12:54:37.843507 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.843483    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cs2d\" (UniqueName: \"kubernetes.io/projected/2dd9546e-8cc6-4066-8aa2-7640b678d59a-kube-api-access-4cs2d\") pod \"managed-serviceaccount-addon-agent-5858659ffd-ds9xt\" (UID: \"2dd9546e-8cc6-4066-8aa2-7640b678d59a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"
Jul 28 12:54:37.931219 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:37.931125    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"
Jul 28 12:54:38.662603 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:38.662579    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt"]
Jul 28 12:54:38.665553 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:38.665529    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dd9546e_8cc6_4066_8aa2_7640b678d59a.slice/crio-0c7ed16838fca2918937381e84915217caf3e6ce5074c412515e0bef0b108e62 WatchSource:0}: Error finding container 0c7ed16838fca2918937381e84915217caf3e6ce5074c412515e0bef0b108e62: Status 404 returned error can't find the container with id 0c7ed16838fca2918937381e84915217caf3e6ce5074c412515e0bef0b108e62
Jul 28 12:54:39.142887 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.142853    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"]
Jul 28 12:54:39.160666 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.160641    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"]
Jul 28 12:54:39.160812 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.160735    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"
Jul 28 12:54:39.163298 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.163276    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hnpsk\""
Jul 28 12:54:39.163403 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.163359    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 12:54:39.339301 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.339260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/04bddd1a-5db7-46dc-a8bc-f725240331e9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2jxtg\" (UID: \"04bddd1a-5db7-46dc-a8bc-f725240331e9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"
Jul 28 12:54:39.440341 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.440259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/04bddd1a-5db7-46dc-a8bc-f725240331e9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2jxtg\" (UID: \"04bddd1a-5db7-46dc-a8bc-f725240331e9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"
Jul 28 12:54:39.444490 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.444466    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/04bddd1a-5db7-46dc-a8bc-f725240331e9-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2jxtg\" (UID: \"04bddd1a-5db7-46dc-a8bc-f725240331e9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"
Jul 28 12:54:39.469581 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.469537    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"
Jul 28 12:54:39.617026 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.616984    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt" event={"ID":"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c","Type":"ContainerStarted","Data":"fe980ff816f68694cfdb8fc2dbd6726ce231423e5d20e5c5c63bb99d0e0b2cda"}
Jul 28 12:54:39.617224 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.617055    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt" event={"ID":"9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c","Type":"ContainerStarted","Data":"6becf00549ab5a958b9be58f634e8df94ca444aead8d8d676301f5ccc516d83a"}
Jul 28 12:54:39.619883 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.619718    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5" event={"ID":"25ff57f5-17a0-46b8-9fa6-6e6a05de1ded","Type":"ContainerStarted","Data":"7d3587dc354ba636bb7fe5fd2a970afeebd4772549b288e75599e7428240b521"}
Jul 28 12:54:39.621860 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.621813    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt" event={"ID":"2dd9546e-8cc6-4066-8aa2-7640b678d59a","Type":"ContainerStarted","Data":"0c7ed16838fca2918937381e84915217caf3e6ce5074c412515e0bef0b108e62"}
Jul 28 12:54:39.623796 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.623770    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx" event={"ID":"4c4f466e-4360-449f-8ab8-da1732648943","Type":"ContainerStarted","Data":"4f0b0c19197a0eb1d18cc93bffd89e4c1d395158604599a2e03ed71c2392dab7"}
Jul 28 12:54:39.638473 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.638423    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-4wxjt" podStartSLOduration=34.314434596 podStartE2EDuration="37.638404014s" totalImagesPullingTime="3.323969418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:39.6379205 +0000 UTC m=+78.960913490" watchObservedRunningTime="2026-07-28 12:54:39.638404014 +0000 UTC m=+78.961396951"
Jul 28 12:54:39.649462 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.648746    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"]
Jul 28 12:54:39.656321 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:39.656292    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bddd1a_5db7_46dc_a8bc_f725240331e9.slice/crio-ea15cc0b89a66803af13497377ba8bd726bdfda5d8538f255bc8a40c66179950 WatchSource:0}: Error finding container ea15cc0b89a66803af13497377ba8bd726bdfda5d8538f255bc8a40c66179950: Status 404 returned error can't find the container with id ea15cc0b89a66803af13497377ba8bd726bdfda5d8538f255bc8a40c66179950
Jul 28 12:54:39.664832 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.664782    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-9dkd5" podStartSLOduration=34.259938536 podStartE2EDuration="37.664764282s" totalImagesPullingTime="3.404825746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:39.663090546 +0000 UTC m=+78.986083486" watchObservedRunningTime="2026-07-28 12:54:39.664764282 +0000 UTC m=+78.987757224"
Jul 28 12:54:39.683395 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:39.683345    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8kwgx" podStartSLOduration=34.35273076 podStartE2EDuration="37.683330995s" totalImagesPullingTime="3.330600235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:39.680576369 +0000 UTC m=+79.003569316" watchObservedRunningTime="2026-07-28 12:54:39.683330995 +0000 UTC m=+79.006323937"
Jul 28 12:54:40.581829 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:40.581796    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5c2st"
Jul 28 12:54:40.631292 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:40.631238    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg" event={"ID":"04bddd1a-5db7-46dc-a8bc-f725240331e9","Type":"ContainerStarted","Data":"ea15cc0b89a66803af13497377ba8bd726bdfda5d8538f255bc8a40c66179950"}
Jul 28 12:54:42.640272 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:42.640227    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt" event={"ID":"2dd9546e-8cc6-4066-8aa2-7640b678d59a","Type":"ContainerStarted","Data":"a230b9f2993f75c0b686834b6ffe89307104e21a315117e021108bf4c160d4c1"}
Jul 28 12:54:42.641672 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:42.641645    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg" event={"ID":"04bddd1a-5db7-46dc-a8bc-f725240331e9","Type":"ContainerStarted","Data":"5670250ecda78b6e8e564768628b80222d369aff73fa2c941a08f5939f9d8702"}
Jul 28 12:54:42.641852 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:42.641838    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"
Jul 28 12:54:42.646657 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:42.646634    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg"
Jul 28 12:54:42.657188 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:42.657150    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5858659ffd-ds9xt" podStartSLOduration=2.406640756 podStartE2EDuration="5.657134356s" totalImagesPullingTime="3.2504936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:42.655570278 +0000 UTC m=+81.978563219" watchObservedRunningTime="2026-07-28 12:54:42.657134356 +0000 UTC m=+81.980127295"
Jul 28 12:54:42.673423 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:42.673377    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2jxtg" podStartSLOduration=1.411835489 podStartE2EDuration="3.673363814s" totalImagesPullingTime="2.261528325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:42.672853503 +0000 UTC m=+81.995846443" watchObservedRunningTime="2026-07-28 12:54:42.673363814 +0000 UTC m=+81.996356753"
Jul 28 12:54:43.882038 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:43.881912    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:43.884195 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:43.884173    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08ef64bf-f32f-4215-8259-6a70cf9f3f84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcbtm\" (UID: \"08ef64bf-f32f-4215-8259-6a70cf9f3f84\") " pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:44.180242 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:44.180155    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-blfqk\""
Jul 28 12:54:44.188015 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:44.187988    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lcbtm"
Jul 28 12:54:44.319619 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:44.319586    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lcbtm"]
Jul 28 12:54:44.322636 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:44.322606    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ef64bf_f32f_4215_8259_6a70cf9f3f84.slice/crio-fee001be62a1520534b6d9be9538fc8a066fc2dca27a1a34cb7ad1f9cac4d354 WatchSource:0}: Error finding container fee001be62a1520534b6d9be9538fc8a066fc2dca27a1a34cb7ad1f9cac4d354: Status 404 returned error can't find the container with id fee001be62a1520534b6d9be9538fc8a066fc2dca27a1a34cb7ad1f9cac4d354
Jul 28 12:54:44.650829 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:44.650790    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcbtm" event={"ID":"08ef64bf-f32f-4215-8259-6a70cf9f3f84","Type":"ContainerStarted","Data":"fee001be62a1520534b6d9be9538fc8a066fc2dca27a1a34cb7ad1f9cac4d354"}
Jul 28 12:54:45.181486 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:45.181456    2572 scope.go:117] "RemoveContainer" containerID="5159f62f34c2695f827f66422fcd7acd5d886bb05b0097a684781b2c19502609"
Jul 28 12:54:45.181955 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:45.181617    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-zmt5l_openshift-console-operator(289ee42c-3969-4bcb-90f5-c0fe0ea9562f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" podUID="289ee42c-3969-4bcb-90f5-c0fe0ea9562f"
Jul 28 12:54:47.636596 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:47.636547    2572 patch_prober.go:28] interesting pod/image-registry-b4bf4b95c-pzrf9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 12:54:47.637062 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:47.636621    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9" podUID="0435f7c0-0018-49a2-b8c1-c4048884fc03" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:54:49.573067 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:49.573040    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:54:49.673659 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:49.673623    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcbtm" event={"ID":"08ef64bf-f32f-4215-8259-6a70cf9f3f84","Type":"ContainerStarted","Data":"16d73b05ba21361890dc042122b2ca8a1162981302f09c6ba58558deefddf503"}
Jul 28 12:54:51.629988 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.629937    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9k8tg"]
Jul 28 12:54:51.642926 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.642888    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.646045 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.645781    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:54:51.646045 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.645936    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:54:51.646045 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.645960    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:54:51.646303 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.646283    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:54:51.646587 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.646558    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bjvgr\""
Jul 28 12:54:51.647405 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.647208    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5kv6\" (UniqueName: \"kubernetes.io/projected/79b89a03-eb00-4600-8dbb-14315a23d3a3-kube-api-access-b5kv6\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.647405 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.647263    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.647405 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.647324    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79b89a03-eb00-4600-8dbb-14315a23d3a3-sys\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.647405 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.647349    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79b89a03-eb00-4600-8dbb-14315a23d3a3-root\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.647675 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.647449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-wtmp\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.647675 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.647494    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.647675 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.647526    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b89a03-eb00-4600-8dbb-14315a23d3a3-metrics-client-ca\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.647675 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.647560    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-textfile\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.647675 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.647596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-tls\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748318 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748279    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5kv6\" (UniqueName: \"kubernetes.io/projected/79b89a03-eb00-4600-8dbb-14315a23d3a3-kube-api-access-b5kv6\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748318 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748325    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748586 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748370    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79b89a03-eb00-4600-8dbb-14315a23d3a3-sys\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748586 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79b89a03-eb00-4600-8dbb-14315a23d3a3-root\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748586 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748439    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-wtmp\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748586 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748450    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79b89a03-eb00-4600-8dbb-14315a23d3a3-sys\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748586 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748466    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748586 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748505    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b89a03-eb00-4600-8dbb-14315a23d3a3-metrics-client-ca\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748586 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748519    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79b89a03-eb00-4600-8dbb-14315a23d3a3-root\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748586 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748543    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-textfile\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.748586 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748587    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-tls\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.749017 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:51.748693    2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 12:54:51.749017 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:51.748749    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-tls podName:79b89a03-eb00-4600-8dbb-14315a23d3a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:52.248729673 +0000 UTC m=+91.571722590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-tls") pod "node-exporter-9k8tg" (UID: "79b89a03-eb00-4600-8dbb-14315a23d3a3") : secret "node-exporter-tls" not found
Jul 28 12:54:51.749017 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.748818    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-wtmp\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.749139 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.749037    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.749139 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.749115    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79b89a03-eb00-4600-8dbb-14315a23d3a3-metrics-client-ca\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.749218 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.749144    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-textfile\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.751187 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.751166    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:51.761106 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:51.761078    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5kv6\" (UniqueName: \"kubernetes.io/projected/79b89a03-eb00-4600-8dbb-14315a23d3a3-kube-api-access-b5kv6\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:52.252397 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:52.252358    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-tls\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:52.252573 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:52.252524    2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 12:54:52.252648 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:54:52.252620    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-tls podName:79b89a03-eb00-4600-8dbb-14315a23d3a3 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:53.252600205 +0000 UTC m=+92.575593123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-tls") pod "node-exporter-9k8tg" (UID: "79b89a03-eb00-4600-8dbb-14315a23d3a3") : secret "node-exporter-tls" not found
Jul 28 12:54:53.263641 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:53.263593    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-tls\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:53.266633 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:53.266604    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79b89a03-eb00-4600-8dbb-14315a23d3a3-node-exporter-tls\") pod \"node-exporter-9k8tg\" (UID: \"79b89a03-eb00-4600-8dbb-14315a23d3a3\") " pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:53.456152 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:53.456113    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9k8tg"
Jul 28 12:54:53.470805 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:54:53.470772    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b89a03_eb00_4600_8dbb_14315a23d3a3.slice/crio-36403c8a094e1dc81bf7f67574228ba1529e787bfb3d42f42ea47544aa5068c7 WatchSource:0}: Error finding container 36403c8a094e1dc81bf7f67574228ba1529e787bfb3d42f42ea47544aa5068c7: Status 404 returned error can't find the container with id 36403c8a094e1dc81bf7f67574228ba1529e787bfb3d42f42ea47544aa5068c7
Jul 28 12:54:53.693752 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:53.693657    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9k8tg" event={"ID":"79b89a03-eb00-4600-8dbb-14315a23d3a3","Type":"ContainerStarted","Data":"36403c8a094e1dc81bf7f67574228ba1529e787bfb3d42f42ea47544aa5068c7"}
Jul 28 12:54:53.695693 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:53.695659    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcbtm" event={"ID":"08ef64bf-f32f-4215-8259-6a70cf9f3f84","Type":"ContainerStarted","Data":"a773b4b4a3580ac780e3e26933f5e836f6bcda59d93012fde6cc600ccc6245c0"}
Jul 28 12:54:54.701553 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:54.701464    2572 generic.go:356] "Generic (PLEG): container finished" podID="79b89a03-eb00-4600-8dbb-14315a23d3a3" containerID="06ccd874e4be6a71398cba71cf08901e3d583fff2b898fe703b2482bfe9c67c0" exitCode=0
Jul 28 12:54:54.702029 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:54.701557    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9k8tg" event={"ID":"79b89a03-eb00-4600-8dbb-14315a23d3a3","Type":"ContainerDied","Data":"06ccd874e4be6a71398cba71cf08901e3d583fff2b898fe703b2482bfe9c67c0"}
Jul 28 12:54:56.175843 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.175755    2572 scope.go:117] "RemoveContainer" containerID="5159f62f34c2695f827f66422fcd7acd5d886bb05b0097a684781b2c19502609"
Jul 28 12:54:56.711834 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.711799    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9k8tg" event={"ID":"79b89a03-eb00-4600-8dbb-14315a23d3a3","Type":"ContainerStarted","Data":"6d8d702cdd27d1cc0700c259fc01f69c6edd5820d2e78c3c817a9742c13931d1"}
Jul 28 12:54:56.712059 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.711866    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9k8tg" event={"ID":"79b89a03-eb00-4600-8dbb-14315a23d3a3","Type":"ContainerStarted","Data":"2a1ebe67e0933ca321593b581e2e4605588b1ba5dc9231d6e1baf144814fd21c"}
Jul 28 12:54:56.713887 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.713858    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcbtm" event={"ID":"08ef64bf-f32f-4215-8259-6a70cf9f3f84","Type":"ContainerStarted","Data":"d764f5a971887629b62eb3b15c1e76d32d3b32882cc6684924fc02db7b683bed"}
Jul 28 12:54:56.715582 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.715564    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 12:54:56.715674 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.715644    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" event={"ID":"289ee42c-3969-4bcb-90f5-c0fe0ea9562f","Type":"ContainerStarted","Data":"e8c486dfa7f00b9db28985f766d02db2a1ca27e122e2fe57464b3a024a964256"}
Jul 28 12:54:56.715916 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.715896    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:56.733478 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.733425    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9k8tg" podStartSLOduration=4.776665703 podStartE2EDuration="5.733411408s" totalImagesPullingTime="880.623495ms" totalInitContainerRuntime="76.12221ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:56.731588515 +0000 UTC m=+96.054581458" watchObservedRunningTime="2026-07-28 12:54:56.733411408 +0000 UTC m=+96.056404366"
Jul 28 12:54:56.751508 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.751451    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lcbtm" podStartSLOduration=32.863070064 podStartE2EDuration="44.751431296s" totalImagesPullingTime="11.888361232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:54:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:56.751208128 +0000 UTC m=+96.074201069" watchObservedRunningTime="2026-07-28 12:54:56.751431296 +0000 UTC m=+96.074424237"
Jul 28 12:54:56.770510 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:56.770464    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l" podStartSLOduration=47.876860016 podStartE2EDuration="54.770449076s" totalImagesPullingTime="6.89358906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:56.769586021 +0000 UTC m=+96.092578962" watchObservedRunningTime="2026-07-28 12:54:56.770449076 +0000 UTC m=+96.093442030"
Jul 28 12:54:57.144468 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:57.144434    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-zmt5l"
Jul 28 12:54:57.550384 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:54:57.550347    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lkksh"
Jul 28 12:55:01.181231 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:01.181149    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b4bf4b95c-pzrf9"]
Jul 28 12:55:16.788008 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:16.787964    2572 generic.go:356] "Generic (PLEG): container finished" podID="8f6ec1da-be3a-4780-9199-dc20768d96e2" containerID="040a40f1418ec8894ae8d042847f453479f44b04e34894b9935377417d03c5d8" exitCode=0
Jul 28 12:55:16.788382 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:16.788033    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rgl5v" event={"ID":"8f6ec1da-be3a-4780-9199-dc20768d96e2","Type":"ContainerDied","Data":"040a40f1418ec8894ae8d042847f453479f44b04e34894b9935377417d03c5d8"}
Jul 28 12:55:16.788382 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:16.788361    2572 scope.go:117] "RemoveContainer" containerID="040a40f1418ec8894ae8d042847f453479f44b04e34894b9935377417d03c5d8"
Jul 28 12:55:17.793818 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:17.793786    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rgl5v" event={"ID":"8f6ec1da-be3a-4780-9199-dc20768d96e2","Type":"ContainerStarted","Data":"91cbeff7f610a937afbea5ec6743227144d4e9317e9c2aa6d00d80054a872bcf"}
Jul 28 12:55:26.200131 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.200041    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9" podUID="0435f7c0-0018-49a2-b8c1-c4048884fc03" containerName="registry" containerID="cri-o://d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866" gracePeriod=30
Jul 28 12:55:26.456325 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.456262    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:55:26.560790 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.560750    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-certificates\") pod \"0435f7c0-0018-49a2-b8c1-c4048884fc03\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") "
Jul 28 12:55:26.561010 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.560803    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48spq\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-kube-api-access-48spq\") pod \"0435f7c0-0018-49a2-b8c1-c4048884fc03\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") "
Jul 28 12:55:26.561010 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.560829    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-installation-pull-secrets\") pod \"0435f7c0-0018-49a2-b8c1-c4048884fc03\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") "
Jul 28 12:55:26.561010 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.560870    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-trusted-ca\") pod \"0435f7c0-0018-49a2-b8c1-c4048884fc03\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") "
Jul 28 12:55:26.561010 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.560900    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") pod \"0435f7c0-0018-49a2-b8c1-c4048884fc03\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") "
Jul 28 12:55:26.561010 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.560928    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-image-registry-private-configuration\") pod \"0435f7c0-0018-49a2-b8c1-c4048884fc03\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") "
Jul 28 12:55:26.561010 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.560983    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-bound-sa-token\") pod \"0435f7c0-0018-49a2-b8c1-c4048884fc03\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") "
Jul 28 12:55:26.561313 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.561031    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0435f7c0-0018-49a2-b8c1-c4048884fc03-ca-trust-extracted\") pod \"0435f7c0-0018-49a2-b8c1-c4048884fc03\" (UID: \"0435f7c0-0018-49a2-b8c1-c4048884fc03\") "
Jul 28 12:55:26.561409 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.561377    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0435f7c0-0018-49a2-b8c1-c4048884fc03" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:55:26.561732 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.561709    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0435f7c0-0018-49a2-b8c1-c4048884fc03" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:55:26.563601 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.563569    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0435f7c0-0018-49a2-b8c1-c4048884fc03" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:55:26.563716 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.563651    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0435f7c0-0018-49a2-b8c1-c4048884fc03" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:55:26.563716 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.563697    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0435f7c0-0018-49a2-b8c1-c4048884fc03" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:55:26.563807 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.563731    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0435f7c0-0018-49a2-b8c1-c4048884fc03" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:55:26.563842 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.563813    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-kube-api-access-48spq" (OuterVolumeSpecName: "kube-api-access-48spq") pod "0435f7c0-0018-49a2-b8c1-c4048884fc03" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03"). InnerVolumeSpecName "kube-api-access-48spq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:55:26.569585 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.569562    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0435f7c0-0018-49a2-b8c1-c4048884fc03-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0435f7c0-0018-49a2-b8c1-c4048884fc03" (UID: "0435f7c0-0018-49a2-b8c1-c4048884fc03"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:55:26.662237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.662185    2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0435f7c0-0018-49a2-b8c1-c4048884fc03-ca-trust-extracted\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 12:55:26.662237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.662229    2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-certificates\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 12:55:26.662237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.662239    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48spq\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-kube-api-access-48spq\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 12:55:26.662237 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.662249    2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-installation-pull-secrets\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 12:55:26.662516 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.662259    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0435f7c0-0018-49a2-b8c1-c4048884fc03-trusted-ca\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 12:55:26.662516 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.662268    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-registry-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 12:55:26.662516 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.662277    2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0435f7c0-0018-49a2-b8c1-c4048884fc03-image-registry-private-configuration\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 12:55:26.662516 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.662286    2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0435f7c0-0018-49a2-b8c1-c4048884fc03-bound-sa-token\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 12:55:26.827920 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.827889    2572 generic.go:356] "Generic (PLEG): container finished" podID="0435f7c0-0018-49a2-b8c1-c4048884fc03" containerID="d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866" exitCode=0
Jul 28 12:55:26.828112 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.827964    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9"
Jul 28 12:55:26.828112 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.827985    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9" event={"ID":"0435f7c0-0018-49a2-b8c1-c4048884fc03","Type":"ContainerDied","Data":"d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866"}
Jul 28 12:55:26.828112 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.828039    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4bf4b95c-pzrf9" event={"ID":"0435f7c0-0018-49a2-b8c1-c4048884fc03","Type":"ContainerDied","Data":"1df855f8c173dd8c304921448c54fc18cb232dacdfba0de6e9e01cc5917f6831"}
Jul 28 12:55:26.828112 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.828066    2572 scope.go:117] "RemoveContainer" containerID="d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866"
Jul 28 12:55:26.829625 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.829597    2572 generic.go:356] "Generic (PLEG): container finished" podID="cb4c7102-dee6-436c-8d1a-3e4b83206308" containerID="f8d7d4a745358d153906f5d2213391bc1c72673146490b00ebcf30b65187f3c9" exitCode=0
Jul 28 12:55:26.829731 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.829646    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv" event={"ID":"cb4c7102-dee6-436c-8d1a-3e4b83206308","Type":"ContainerDied","Data":"f8d7d4a745358d153906f5d2213391bc1c72673146490b00ebcf30b65187f3c9"}
Jul 28 12:55:26.830132 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.830106    2572 scope.go:117] "RemoveContainer" containerID="f8d7d4a745358d153906f5d2213391bc1c72673146490b00ebcf30b65187f3c9"
Jul 28 12:55:26.839431 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.839415    2572 scope.go:117] "RemoveContainer" containerID="d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866"
Jul 28 12:55:26.839645 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:55:26.839624    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866\": container with ID starting with d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866 not found: ID does not exist" containerID="d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866"
Jul 28 12:55:26.839702 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.839651    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866"} err="failed to get container status \"d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866\": rpc error: code = NotFound desc = could not find container \"d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866\": container with ID starting with d3721629c80b286f3c59ddefadb98e0c7609c6e2a41af693813eccb28463b866 not found: ID does not exist"
Jul 28 12:55:26.870771 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.870743    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b4bf4b95c-pzrf9"]
Jul 28 12:55:26.877908 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:26.877883    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b4bf4b95c-pzrf9"]
Jul 28 12:55:27.184255 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:27.182074    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0435f7c0-0018-49a2-b8c1-c4048884fc03" path="/var/lib/kubelet/pods/0435f7c0-0018-49a2-b8c1-c4048884fc03/volumes"
Jul 28 12:55:27.835305 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:27.835266    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-45mdv" event={"ID":"cb4c7102-dee6-436c-8d1a-3e4b83206308","Type":"ContainerStarted","Data":"c5612ebd7896ba1b82357a884cdb25d1343855ce030ef300acfd536047a405c6"}
Jul 28 12:55:29.992669 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:29.992615    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:55:29.995115 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:29.995090    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fa5143-75d6-4f79-88d0-97b5a5ecf09c-metrics-certs\") pod \"network-metrics-daemon-k4prs\" (UID: \"43fa5143-75d6-4f79-88d0-97b5a5ecf09c\") " pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:55:30.197273 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:30.197244    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z7m8f\""
Jul 28 12:55:30.206610 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:30.206327    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4prs"
Jul 28 12:55:30.383591 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:30.383557    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k4prs"]
Jul 28 12:55:30.386536 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:55:30.386508    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fa5143_75d6_4f79_88d0_97b5a5ecf09c.slice/crio-e0a79f3f008e9b9a118d82ce8dcd7a1e881badfbb504a2df96fd1a9d0827ddca WatchSource:0}: Error finding container e0a79f3f008e9b9a118d82ce8dcd7a1e881badfbb504a2df96fd1a9d0827ddca: Status 404 returned error can't find the container with id e0a79f3f008e9b9a118d82ce8dcd7a1e881badfbb504a2df96fd1a9d0827ddca
Jul 28 12:55:30.851301 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:30.851262    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k4prs" event={"ID":"43fa5143-75d6-4f79-88d0-97b5a5ecf09c","Type":"ContainerStarted","Data":"e0a79f3f008e9b9a118d82ce8dcd7a1e881badfbb504a2df96fd1a9d0827ddca"}
Jul 28 12:55:30.852828 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:30.852747    2572 generic.go:356] "Generic (PLEG): container finished" podID="fd4afe72-14f5-4172-b6e9-3eee63e1c5f6" containerID="fab670efa39560444efb173ea6cfd6c73eb5429e6aba7e0c15919877e9bee02c" exitCode=0
Jul 28 12:55:30.852828 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:30.852818    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj" event={"ID":"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6","Type":"ContainerDied","Data":"fab670efa39560444efb173ea6cfd6c73eb5429e6aba7e0c15919877e9bee02c"}
Jul 28 12:55:30.853210 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:30.853190    2572 scope.go:117] "RemoveContainer" containerID="fab670efa39560444efb173ea6cfd6c73eb5429e6aba7e0c15919877e9bee02c"
Jul 28 12:55:31.858576 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:31.858540    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k4prs" event={"ID":"43fa5143-75d6-4f79-88d0-97b5a5ecf09c","Type":"ContainerStarted","Data":"66ef1dc51504c29f2febcd98f9255a4909b46622579cc856ab802ead822689f4"}
Jul 28 12:55:31.859104 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:31.858598    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k4prs" event={"ID":"43fa5143-75d6-4f79-88d0-97b5a5ecf09c","Type":"ContainerStarted","Data":"04760555eb1598420bbb8293ee9f8910a718e73cb47784a5c9f728d8a38322e1"}
Jul 28 12:55:31.860332 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:31.860307    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj" event={"ID":"fd4afe72-14f5-4172-b6e9-3eee63e1c5f6","Type":"ContainerStarted","Data":"fe53c43616c22c39365567306bff3803fc7808d17fd00420e0b986df79641c1a"}
Jul 28 12:55:31.876007 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:55:31.875925    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k4prs" podStartSLOduration=129.947051045 podStartE2EDuration="2m10.875907075s" totalImagesPullingTime="928.85603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:31.875162139 +0000 UTC m=+131.198155079" watchObservedRunningTime="2026-07-28 12:55:31.875907075 +0000 UTC m=+131.198900015"
Jul 28 12:58:21.045684 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:58:21.045653    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 12:58:21.061732 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:58:21.061703    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 12:58:21.062014 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:58:21.061990    2572 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 12:59:41.648000 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.647917    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"]
Jul 28 12:59:41.648484 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.648284    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0435f7c0-0018-49a2-b8c1-c4048884fc03" containerName="registry"
Jul 28 12:59:41.648484 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.648300    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0435f7c0-0018-49a2-b8c1-c4048884fc03" containerName="registry"
Jul 28 12:59:41.648484 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.648360    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0435f7c0-0018-49a2-b8c1-c4048884fc03" containerName="registry"
Jul 28 12:59:41.651338 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.651318    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 12:59:41.655085 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.655063    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 12:59:41.655395 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.655366    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-wt4f8\""
Jul 28 12:59:41.655395 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.655386    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 12:59:41.655485 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.655462    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 12:59:41.672981 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.672956    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"]
Jul 28 12:59:41.713234 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.713201    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5c12f765-8573-4ddb-b68e-e8dcd67c1efc-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-75dpw\" (UID: \"5c12f765-8573-4ddb-b68e-e8dcd67c1efc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 12:59:41.713391 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.713241    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswn5\" (UniqueName: \"kubernetes.io/projected/5c12f765-8573-4ddb-b68e-e8dcd67c1efc-kube-api-access-wswn5\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-75dpw\" (UID: \"5c12f765-8573-4ddb-b68e-e8dcd67c1efc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 12:59:41.813913 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.813880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5c12f765-8573-4ddb-b68e-e8dcd67c1efc-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-75dpw\" (UID: \"5c12f765-8573-4ddb-b68e-e8dcd67c1efc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 12:59:41.814100 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.813924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswn5\" (UniqueName: \"kubernetes.io/projected/5c12f765-8573-4ddb-b68e-e8dcd67c1efc-kube-api-access-wswn5\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-75dpw\" (UID: \"5c12f765-8573-4ddb-b68e-e8dcd67c1efc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 12:59:41.816351 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.816318    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5c12f765-8573-4ddb-b68e-e8dcd67c1efc-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-75dpw\" (UID: \"5c12f765-8573-4ddb-b68e-e8dcd67c1efc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 12:59:41.823169 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.823149    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswn5\" (UniqueName: \"kubernetes.io/projected/5c12f765-8573-4ddb-b68e-e8dcd67c1efc-kube-api-access-wswn5\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-75dpw\" (UID: \"5c12f765-8573-4ddb-b68e-e8dcd67c1efc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 12:59:41.961486 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:41.961384    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 12:59:42.107378 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:42.107352    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"]
Jul 28 12:59:42.110343 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:59:42.110315    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c12f765_8573_4ddb_b68e_e8dcd67c1efc.slice/crio-45bee5fa0afa5f2fb372bfe1be840141a5aef837793265e81b48ec7e895317f4 WatchSource:0}: Error finding container 45bee5fa0afa5f2fb372bfe1be840141a5aef837793265e81b48ec7e895317f4: Status 404 returned error can't find the container with id 45bee5fa0afa5f2fb372bfe1be840141a5aef837793265e81b48ec7e895317f4
Jul 28 12:59:42.112350 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:42.112330    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:59:42.793613 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:42.793580    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw" event={"ID":"5c12f765-8573-4ddb-b68e-e8dcd67c1efc","Type":"ContainerStarted","Data":"45bee5fa0afa5f2fb372bfe1be840141a5aef837793265e81b48ec7e895317f4"}
Jul 28 12:59:45.810888 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:45.810849    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw" event={"ID":"5c12f765-8573-4ddb-b68e-e8dcd67c1efc","Type":"ContainerStarted","Data":"358e944d5eb756b78ff8a8f5747446acb6aba9cea0d47c227af495ce8e172726"}
Jul 28 12:59:45.811315 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:45.810985    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 12:59:45.835756 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:45.835696    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw" podStartSLOduration=1.598785476 podStartE2EDuration="4.835680394s" totalImagesPullingTime="3.236894918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:59:45.83397864 +0000 UTC m=+385.156971581" watchObservedRunningTime="2026-07-28 12:59:45.835680394 +0000 UTC m=+385.158673333"
Jul 28 12:59:46.100399 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.100313    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-7p8km"]
Jul 28 12:59:46.103608 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.103591    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.105883 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.105854    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 12:59:46.106037 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.105863    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-l6c5l\""
Jul 28 12:59:46.106037 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.105863    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 12:59:46.115775 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.115752    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-7p8km"]
Jul 28 12:59:46.149235 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.149203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-certificates\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.149235 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.149237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqp2\" (UniqueName: \"kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-kube-api-access-fpqp2\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.149447 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.149260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-cabundle0\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.249869 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.249829    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-cabundle0\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.250098 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.249937    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-certificates\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.250098 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.249988    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqp2\" (UniqueName: \"kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-kube-api-access-fpqp2\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.250225 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:59:46.250114    2572 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 12:59:46.250225 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:59:46.250140    2572 secret.go:281] references non-existent secret key: ca.crt
Jul 28 12:59:46.250225 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:59:46.250152    2572 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 12:59:46.250225 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:59:46.250170    2572 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-7p8km: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:59:46.250385 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:59:46.250242    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-certificates podName:71dc66e3-73b8-4512-b7c4-88c1a4fe12eb nodeName:}" failed. No retries permitted until 2026-07-28 12:59:46.750221021 +0000 UTC m=+386.073213961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-certificates") pod "keda-operator-77f648d694-7p8km" (UID: "71dc66e3-73b8-4512-b7c4-88c1a4fe12eb") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:59:46.250702 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.250681    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-cabundle0\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.260507 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.260482    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqp2\" (UniqueName: \"kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-kube-api-access-fpqp2\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.754348 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.754309    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-certificates\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:46.754535 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:59:46.754460    2572 secret.go:281] references non-existent secret key: ca.crt
Jul 28 12:59:46.754535 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:59:46.754475    2572 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 12:59:46.754535 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:59:46.754484    2572 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-7p8km: references non-existent secret key: ca.crt
Jul 28 12:59:46.754635 ip-10-0-142-83 kubenswrapper[2572]: E0728 12:59:46.754551    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-certificates podName:71dc66e3-73b8-4512-b7c4-88c1a4fe12eb nodeName:}" failed. No retries permitted until 2026-07-28 12:59:47.754531703 +0000 UTC m=+387.077524623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-certificates") pod "keda-operator-77f648d694-7p8km" (UID: "71dc66e3-73b8-4512-b7c4-88c1a4fe12eb") : references non-existent secret key: ca.crt
Jul 28 12:59:46.980587 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.980553    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"]
Jul 28 12:59:46.983844 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.983828    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:46.986325 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.986305    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 12:59:46.997383 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:46.997356    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"]
Jul 28 12:59:47.057784 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.057749    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vwwt\" (UniqueName: \"kubernetes.io/projected/9a9ca898-2261-43f7-aa9a-f6670379c643-kube-api-access-4vwwt\") pod \"keda-metrics-apiserver-55557df8c-wznsc\" (UID: \"9a9ca898-2261-43f7-aa9a-f6670379c643\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.057980 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.057847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9a9ca898-2261-43f7-aa9a-f6670379c643-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-wznsc\" (UID: \"9a9ca898-2261-43f7-aa9a-f6670379c643\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.057980 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.057870    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9a9ca898-2261-43f7-aa9a-f6670379c643-certificates\") pod \"keda-metrics-apiserver-55557df8c-wznsc\" (UID: \"9a9ca898-2261-43f7-aa9a-f6670379c643\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.158701 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.158667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vwwt\" (UniqueName: \"kubernetes.io/projected/9a9ca898-2261-43f7-aa9a-f6670379c643-kube-api-access-4vwwt\") pod \"keda-metrics-apiserver-55557df8c-wznsc\" (UID: \"9a9ca898-2261-43f7-aa9a-f6670379c643\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.158987 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.158732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9a9ca898-2261-43f7-aa9a-f6670379c643-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-wznsc\" (UID: \"9a9ca898-2261-43f7-aa9a-f6670379c643\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.158987 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.158751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9a9ca898-2261-43f7-aa9a-f6670379c643-certificates\") pod \"keda-metrics-apiserver-55557df8c-wznsc\" (UID: \"9a9ca898-2261-43f7-aa9a-f6670379c643\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.159156 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.159135    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9a9ca898-2261-43f7-aa9a-f6670379c643-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-wznsc\" (UID: \"9a9ca898-2261-43f7-aa9a-f6670379c643\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.161396 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.161374    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9a9ca898-2261-43f7-aa9a-f6670379c643-certificates\") pod \"keda-metrics-apiserver-55557df8c-wznsc\" (UID: \"9a9ca898-2261-43f7-aa9a-f6670379c643\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.184312 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.184285    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vwwt\" (UniqueName: \"kubernetes.io/projected/9a9ca898-2261-43f7-aa9a-f6670379c643-kube-api-access-4vwwt\") pod \"keda-metrics-apiserver-55557df8c-wznsc\" (UID: \"9a9ca898-2261-43f7-aa9a-f6670379c643\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.294048 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.294013    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:47.428361 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.428334    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"]
Jul 28 12:59:47.430566 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:59:47.430539    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9ca898_2261_43f7_aa9a_f6670379c643.slice/crio-449ac2408b3ab5e4dc7ea8083dd2f575864df4c7349923d5c585f9ba725410cc WatchSource:0}: Error finding container 449ac2408b3ab5e4dc7ea8083dd2f575864df4c7349923d5c585f9ba725410cc: Status 404 returned error can't find the container with id 449ac2408b3ab5e4dc7ea8083dd2f575864df4c7349923d5c585f9ba725410cc
Jul 28 12:59:47.763980 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.763935    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-certificates\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:47.766484 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.766465    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/71dc66e3-73b8-4512-b7c4-88c1a4fe12eb-certificates\") pod \"keda-operator-77f648d694-7p8km\" (UID: \"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb\") " pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:47.820385 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.820350    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc" event={"ID":"9a9ca898-2261-43f7-aa9a-f6670379c643","Type":"ContainerStarted","Data":"449ac2408b3ab5e4dc7ea8083dd2f575864df4c7349923d5c585f9ba725410cc"}
Jul 28 12:59:47.913767 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:47.913734    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:48.052513 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:48.052461    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-7p8km"]
Jul 28 12:59:48.055051 ip-10-0-142-83 kubenswrapper[2572]: W0728 12:59:48.055021    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dc66e3_73b8_4512_b7c4_88c1a4fe12eb.slice/crio-298ce0424446d179904571ccc4ae73aa8244460b6027fc4cf34408de6dbe43e6 WatchSource:0}: Error finding container 298ce0424446d179904571ccc4ae73aa8244460b6027fc4cf34408de6dbe43e6: Status 404 returned error can't find the container with id 298ce0424446d179904571ccc4ae73aa8244460b6027fc4cf34408de6dbe43e6
Jul 28 12:59:48.827382 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:48.827338    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-7p8km" event={"ID":"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb","Type":"ContainerStarted","Data":"298ce0424446d179904571ccc4ae73aa8244460b6027fc4cf34408de6dbe43e6"}
Jul 28 12:59:51.842579 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:51.842536    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-7p8km" event={"ID":"71dc66e3-73b8-4512-b7c4-88c1a4fe12eb","Type":"ContainerStarted","Data":"2eb32647d7aa7bc20ca4458c86297bdcc2fd888a3f79e987fb52b94162b0248c"}
Jul 28 12:59:51.843081 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:51.842683    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 12:59:51.860037 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:51.859980    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-7p8km" podStartSLOduration=2.941508396 podStartE2EDuration="5.859964934s" totalImagesPullingTime="2.918456538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:59:51.85941043 +0000 UTC m=+391.182403396" watchObservedRunningTime="2026-07-28 12:59:51.859964934 +0000 UTC m=+391.182957875"
Jul 28 12:59:52.848436 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:52.848400    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc" event={"ID":"9a9ca898-2261-43f7-aa9a-f6670379c643","Type":"ContainerStarted","Data":"2dee4b020cd0f30025bf74fe05321adf2d375ac43b922669bcc4d6ae6944039e"}
Jul 28 12:59:52.848843 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:52.848707    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 12:59:52.866270 ip-10-0-142-83 kubenswrapper[2572]: I0728 12:59:52.866221    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc" podStartSLOduration=1.5303345799999999 podStartE2EDuration="6.866204482s" totalImagesPullingTime="5.335869902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:59:52.865458604 +0000 UTC m=+392.188451544" watchObservedRunningTime="2026-07-28 12:59:52.866204482 +0000 UTC m=+392.189197421"
Jul 28 13:00:03.856782 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:00:03.856747    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-wznsc"
Jul 28 13:00:06.817808 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:00:06.817778    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-75dpw"
Jul 28 13:00:12.850458 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:00:12.850423    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-7p8km"
Jul 28 13:01:06.229923 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.229892    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-twc8v"]
Jul 28 13:01:06.232166 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.232150    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:06.234571 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.234547    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 13:01:06.234700 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.234621    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 13:01:06.234700 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.234640    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 13:01:06.235205 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.235189    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-5whk8\""
Jul 28 13:01:06.248397 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.248375    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-twc8v"]
Jul 28 13:01:06.287958 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.287907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a979d2-680f-416c-8a59-c6d7a304fc3f-cert\") pod \"kserve-controller-manager-ff96d5959-twc8v\" (UID: \"33a979d2-680f-416c-8a59-c6d7a304fc3f\") " pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:06.288112 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.288009    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8kb\" (UniqueName: \"kubernetes.io/projected/33a979d2-680f-416c-8a59-c6d7a304fc3f-kube-api-access-cs8kb\") pod \"kserve-controller-manager-ff96d5959-twc8v\" (UID: \"33a979d2-680f-416c-8a59-c6d7a304fc3f\") " pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:06.389067 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.389033    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a979d2-680f-416c-8a59-c6d7a304fc3f-cert\") pod \"kserve-controller-manager-ff96d5959-twc8v\" (UID: \"33a979d2-680f-416c-8a59-c6d7a304fc3f\") " pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:06.389238 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.389081    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8kb\" (UniqueName: \"kubernetes.io/projected/33a979d2-680f-416c-8a59-c6d7a304fc3f-kube-api-access-cs8kb\") pod \"kserve-controller-manager-ff96d5959-twc8v\" (UID: \"33a979d2-680f-416c-8a59-c6d7a304fc3f\") " pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:06.389238 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:01:06.389177    2572 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 13:01:06.389309 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:01:06.389250    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a979d2-680f-416c-8a59-c6d7a304fc3f-cert podName:33a979d2-680f-416c-8a59-c6d7a304fc3f nodeName:}" failed. No retries permitted until 2026-07-28 13:01:06.889233019 +0000 UTC m=+466.212225941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/33a979d2-680f-416c-8a59-c6d7a304fc3f-cert") pod "kserve-controller-manager-ff96d5959-twc8v" (UID: "33a979d2-680f-416c-8a59-c6d7a304fc3f") : secret "kserve-webhook-server-cert" not found
Jul 28 13:01:06.401228 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.401206    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8kb\" (UniqueName: \"kubernetes.io/projected/33a979d2-680f-416c-8a59-c6d7a304fc3f-kube-api-access-cs8kb\") pod \"kserve-controller-manager-ff96d5959-twc8v\" (UID: \"33a979d2-680f-416c-8a59-c6d7a304fc3f\") " pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:06.894489 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.894449    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a979d2-680f-416c-8a59-c6d7a304fc3f-cert\") pod \"kserve-controller-manager-ff96d5959-twc8v\" (UID: \"33a979d2-680f-416c-8a59-c6d7a304fc3f\") " pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:06.896872 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:06.896851    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a979d2-680f-416c-8a59-c6d7a304fc3f-cert\") pod \"kserve-controller-manager-ff96d5959-twc8v\" (UID: \"33a979d2-680f-416c-8a59-c6d7a304fc3f\") " pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:07.142546 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:07.142496    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:07.276161 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:07.276132    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-twc8v"]
Jul 28 13:01:07.279023 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:01:07.278997    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a979d2_680f_416c_8a59_c6d7a304fc3f.slice/crio-efb9c66f3ffcdfd3559454ffd5680fe831ae6f9712f1ee2726b3992e11a2fbf7 WatchSource:0}: Error finding container efb9c66f3ffcdfd3559454ffd5680fe831ae6f9712f1ee2726b3992e11a2fbf7: Status 404 returned error can't find the container with id efb9c66f3ffcdfd3559454ffd5680fe831ae6f9712f1ee2726b3992e11a2fbf7
Jul 28 13:01:08.151493 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:08.151459    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-twc8v" event={"ID":"33a979d2-680f-416c-8a59-c6d7a304fc3f","Type":"ContainerStarted","Data":"efb9c66f3ffcdfd3559454ffd5680fe831ae6f9712f1ee2726b3992e11a2fbf7"}
Jul 28 13:01:11.167161 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:11.167123    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-twc8v" event={"ID":"33a979d2-680f-416c-8a59-c6d7a304fc3f","Type":"ContainerStarted","Data":"de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a"}
Jul 28 13:01:11.167569 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:11.167234    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:11.184807 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:11.184755    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-twc8v" podStartSLOduration=1.7175752370000001 podStartE2EDuration="5.184738816s" totalImagesPullingTime="3.467163579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:01:11.183775298 +0000 UTC m=+470.506768241" watchObservedRunningTime="2026-07-28 13:01:11.184738816 +0000 UTC m=+470.507731747"
Jul 28 13:01:42.177520 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:42.177481    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:49.477020 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.476988    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-twc8v"]
Jul 28 13:01:49.477463 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.477165    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-ff96d5959-twc8v" podUID="33a979d2-680f-416c-8a59-c6d7a304fc3f" containerName="manager" containerID="cri-o://de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a" gracePeriod=10
Jul 28 13:01:49.503198 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.503173    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-qx77q"]
Jul 28 13:01:49.506142 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.506126    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:01:49.517134 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.517110    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-qx77q"]
Jul 28 13:01:49.536547 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.536515    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50ff5da8-7c20-41a5-9260-6623b5af0118-cert\") pod \"kserve-controller-manager-ff96d5959-qx77q\" (UID: \"50ff5da8-7c20-41a5-9260-6623b5af0118\") " pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:01:49.536663 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.536566    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twv9v\" (UniqueName: \"kubernetes.io/projected/50ff5da8-7c20-41a5-9260-6623b5af0118-kube-api-access-twv9v\") pod \"kserve-controller-manager-ff96d5959-qx77q\" (UID: \"50ff5da8-7c20-41a5-9260-6623b5af0118\") " pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:01:49.637413 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.637377    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50ff5da8-7c20-41a5-9260-6623b5af0118-cert\") pod \"kserve-controller-manager-ff96d5959-qx77q\" (UID: \"50ff5da8-7c20-41a5-9260-6623b5af0118\") " pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:01:49.637577 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.637438    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twv9v\" (UniqueName: \"kubernetes.io/projected/50ff5da8-7c20-41a5-9260-6623b5af0118-kube-api-access-twv9v\") pod \"kserve-controller-manager-ff96d5959-qx77q\" (UID: \"50ff5da8-7c20-41a5-9260-6623b5af0118\") " pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:01:49.639693 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.639667    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50ff5da8-7c20-41a5-9260-6623b5af0118-cert\") pod \"kserve-controller-manager-ff96d5959-qx77q\" (UID: \"50ff5da8-7c20-41a5-9260-6623b5af0118\") " pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:01:49.647512 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.647483    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twv9v\" (UniqueName: \"kubernetes.io/projected/50ff5da8-7c20-41a5-9260-6623b5af0118-kube-api-access-twv9v\") pod \"kserve-controller-manager-ff96d5959-qx77q\" (UID: \"50ff5da8-7c20-41a5-9260-6623b5af0118\") " pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:01:49.722622 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.722596    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:49.737990 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.737926    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs8kb\" (UniqueName: \"kubernetes.io/projected/33a979d2-680f-416c-8a59-c6d7a304fc3f-kube-api-access-cs8kb\") pod \"33a979d2-680f-416c-8a59-c6d7a304fc3f\" (UID: \"33a979d2-680f-416c-8a59-c6d7a304fc3f\") "
Jul 28 13:01:49.737990 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.737984    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a979d2-680f-416c-8a59-c6d7a304fc3f-cert\") pod \"33a979d2-680f-416c-8a59-c6d7a304fc3f\" (UID: \"33a979d2-680f-416c-8a59-c6d7a304fc3f\") "
Jul 28 13:01:49.740498 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.740444    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a979d2-680f-416c-8a59-c6d7a304fc3f-cert" (OuterVolumeSpecName: "cert") pod "33a979d2-680f-416c-8a59-c6d7a304fc3f" (UID: "33a979d2-680f-416c-8a59-c6d7a304fc3f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:01:49.740677 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.740630    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a979d2-680f-416c-8a59-c6d7a304fc3f-kube-api-access-cs8kb" (OuterVolumeSpecName: "kube-api-access-cs8kb") pod "33a979d2-680f-416c-8a59-c6d7a304fc3f" (UID: "33a979d2-680f-416c-8a59-c6d7a304fc3f"). InnerVolumeSpecName "kube-api-access-cs8kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:01:49.838926 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.838893    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs8kb\" (UniqueName: \"kubernetes.io/projected/33a979d2-680f-416c-8a59-c6d7a304fc3f-kube-api-access-cs8kb\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:01:49.838926 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.838922    2572 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a979d2-680f-416c-8a59-c6d7a304fc3f-cert\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:01:49.859821 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.859793    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:01:49.987459 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:49.987344    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-qx77q"]
Jul 28 13:01:49.989765 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:01:49.989708    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ff5da8_7c20_41a5_9260_6623b5af0118.slice/crio-35bf54472103df752140ae67c6fb86e31a382f2bc0483b26fb9207c622781f57 WatchSource:0}: Error finding container 35bf54472103df752140ae67c6fb86e31a382f2bc0483b26fb9207c622781f57: Status 404 returned error can't find the container with id 35bf54472103df752140ae67c6fb86e31a382f2bc0483b26fb9207c622781f57
Jul 28 13:01:50.344338 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.344302    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-qx77q" event={"ID":"50ff5da8-7c20-41a5-9260-6623b5af0118","Type":"ContainerStarted","Data":"35bf54472103df752140ae67c6fb86e31a382f2bc0483b26fb9207c622781f57"}
Jul 28 13:01:50.345479 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.345453    2572 generic.go:356] "Generic (PLEG): container finished" podID="33a979d2-680f-416c-8a59-c6d7a304fc3f" containerID="de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a" exitCode=0
Jul 28 13:01:50.345601 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.345493    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-twc8v"
Jul 28 13:01:50.345601 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.345532    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-twc8v" event={"ID":"33a979d2-680f-416c-8a59-c6d7a304fc3f","Type":"ContainerDied","Data":"de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a"}
Jul 28 13:01:50.345601 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.345581    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-twc8v" event={"ID":"33a979d2-680f-416c-8a59-c6d7a304fc3f","Type":"ContainerDied","Data":"efb9c66f3ffcdfd3559454ffd5680fe831ae6f9712f1ee2726b3992e11a2fbf7"}
Jul 28 13:01:50.345776 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.345603    2572 scope.go:117] "RemoveContainer" containerID="de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a"
Jul 28 13:01:50.353920 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.353905    2572 scope.go:117] "RemoveContainer" containerID="de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a"
Jul 28 13:01:50.354164 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:01:50.354147    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a\": container with ID starting with de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a not found: ID does not exist" containerID="de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a"
Jul 28 13:01:50.354209 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.354172    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a"} err="failed to get container status \"de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a\": rpc error: code = NotFound desc = could not find container \"de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a\": container with ID starting with de2a79ff30e6c13a25f92dbe62275cd337a50f8b1030cea9208b97b2260cc40a not found: ID does not exist"
Jul 28 13:01:50.364664 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.364645    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-twc8v"]
Jul 28 13:01:50.368565 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:50.368544    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-twc8v"]
Jul 28 13:01:51.185123 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:51.185092    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a979d2-680f-416c-8a59-c6d7a304fc3f" path="/var/lib/kubelet/pods/33a979d2-680f-416c-8a59-c6d7a304fc3f/volumes"
Jul 28 13:01:51.351216 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:51.351180    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-qx77q" event={"ID":"50ff5da8-7c20-41a5-9260-6623b5af0118","Type":"ContainerStarted","Data":"ca8b88dbe3d084add83d4b8dba10c1b0f442951d346eda93936e0ab7af6c6e21"}
Jul 28 13:01:51.351375 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:51.351279    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:01:51.368835 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:01:51.368790    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-qx77q" podStartSLOduration=1.9507378640000002 podStartE2EDuration="2.368775919s" totalImagesPullingTime="418.038055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:01:51.368374418 +0000 UTC m=+510.691367359" watchObservedRunningTime="2026-07-28 13:01:51.368775919 +0000 UTC m=+510.691768859"
Jul 28 13:02:22.362080 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:22.362045    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-qx77q"
Jul 28 13:02:23.611487 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.611452    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-qr44z"]
Jul 28 13:02:23.611850 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.611766    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a979d2-680f-416c-8a59-c6d7a304fc3f" containerName="manager"
Jul 28 13:02:23.611850 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.611777    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a979d2-680f-416c-8a59-c6d7a304fc3f" containerName="manager"
Jul 28 13:02:23.611850 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.611829    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a979d2-680f-416c-8a59-c6d7a304fc3f" containerName="manager"
Jul 28 13:02:23.615763 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.615746    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:23.618347 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.618325    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-tts8t\""
Jul 28 13:02:23.618452 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.618395    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 13:02:23.625012 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.624991    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-qr44z"]
Jul 28 13:02:23.627795 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.627771    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-vcf8f"]
Jul 28 13:02:23.631307 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.631288    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:23.633934 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.633913    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 13:02:23.634051 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.633988    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-7wf92\""
Jul 28 13:02:23.641588 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.641565    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-vcf8f"]
Jul 28 13:02:23.703626 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.703594    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n4hs\" (UniqueName: \"kubernetes.io/projected/b9369769-2bd9-4aba-b5b7-5d1e052ead91-kube-api-access-5n4hs\") pod \"odh-model-controller-75d9cdbddf-vcf8f\" (UID: \"b9369769-2bd9-4aba-b5b7-5d1e052ead91\") " pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:23.703804 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.703649    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9369769-2bd9-4aba-b5b7-5d1e052ead91-cert\") pod \"odh-model-controller-75d9cdbddf-vcf8f\" (UID: \"b9369769-2bd9-4aba-b5b7-5d1e052ead91\") " pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:23.703804 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.703730    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d72388a7-93f1-42dc-aa06-2aac5e985452-tls-certs\") pod \"model-serving-api-7b954ffc95-qr44z\" (UID: \"d72388a7-93f1-42dc-aa06-2aac5e985452\") " pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:23.703804 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.703766    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27cs\" (UniqueName: \"kubernetes.io/projected/d72388a7-93f1-42dc-aa06-2aac5e985452-kube-api-access-z27cs\") pod \"model-serving-api-7b954ffc95-qr44z\" (UID: \"d72388a7-93f1-42dc-aa06-2aac5e985452\") " pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:23.804906 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.804872    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n4hs\" (UniqueName: \"kubernetes.io/projected/b9369769-2bd9-4aba-b5b7-5d1e052ead91-kube-api-access-5n4hs\") pod \"odh-model-controller-75d9cdbddf-vcf8f\" (UID: \"b9369769-2bd9-4aba-b5b7-5d1e052ead91\") " pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:23.805101 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.804928    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9369769-2bd9-4aba-b5b7-5d1e052ead91-cert\") pod \"odh-model-controller-75d9cdbddf-vcf8f\" (UID: \"b9369769-2bd9-4aba-b5b7-5d1e052ead91\") " pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:23.805101 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.804966    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d72388a7-93f1-42dc-aa06-2aac5e985452-tls-certs\") pod \"model-serving-api-7b954ffc95-qr44z\" (UID: \"d72388a7-93f1-42dc-aa06-2aac5e985452\") " pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:23.805101 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.804983    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27cs\" (UniqueName: \"kubernetes.io/projected/d72388a7-93f1-42dc-aa06-2aac5e985452-kube-api-access-z27cs\") pod \"model-serving-api-7b954ffc95-qr44z\" (UID: \"d72388a7-93f1-42dc-aa06-2aac5e985452\") " pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:23.805101 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:02:23.805084    2572 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 13:02:23.805286 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:02:23.805099    2572 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 13:02:23.805286 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:02:23.805131    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d72388a7-93f1-42dc-aa06-2aac5e985452-tls-certs podName:d72388a7-93f1-42dc-aa06-2aac5e985452 nodeName:}" failed. No retries permitted until 2026-07-28 13:02:24.305115244 +0000 UTC m=+543.628108162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d72388a7-93f1-42dc-aa06-2aac5e985452-tls-certs") pod "model-serving-api-7b954ffc95-qr44z" (UID: "d72388a7-93f1-42dc-aa06-2aac5e985452") : secret "model-serving-api-tls" not found
Jul 28 13:02:23.805286 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:02:23.805157    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9369769-2bd9-4aba-b5b7-5d1e052ead91-cert podName:b9369769-2bd9-4aba-b5b7-5d1e052ead91 nodeName:}" failed. No retries permitted until 2026-07-28 13:02:24.305139539 +0000 UTC m=+543.628132457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b9369769-2bd9-4aba-b5b7-5d1e052ead91-cert") pod "odh-model-controller-75d9cdbddf-vcf8f" (UID: "b9369769-2bd9-4aba-b5b7-5d1e052ead91") : secret "odh-model-controller-webhook-cert" not found
Jul 28 13:02:23.816606 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.816579    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n4hs\" (UniqueName: \"kubernetes.io/projected/b9369769-2bd9-4aba-b5b7-5d1e052ead91-kube-api-access-5n4hs\") pod \"odh-model-controller-75d9cdbddf-vcf8f\" (UID: \"b9369769-2bd9-4aba-b5b7-5d1e052ead91\") " pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:23.818873 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:23.818856    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27cs\" (UniqueName: \"kubernetes.io/projected/d72388a7-93f1-42dc-aa06-2aac5e985452-kube-api-access-z27cs\") pod \"model-serving-api-7b954ffc95-qr44z\" (UID: \"d72388a7-93f1-42dc-aa06-2aac5e985452\") " pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:24.310223 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:24.310188    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9369769-2bd9-4aba-b5b7-5d1e052ead91-cert\") pod \"odh-model-controller-75d9cdbddf-vcf8f\" (UID: \"b9369769-2bd9-4aba-b5b7-5d1e052ead91\") " pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:24.310223 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:24.310231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d72388a7-93f1-42dc-aa06-2aac5e985452-tls-certs\") pod \"model-serving-api-7b954ffc95-qr44z\" (UID: \"d72388a7-93f1-42dc-aa06-2aac5e985452\") " pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:24.312606 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:24.312581    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9369769-2bd9-4aba-b5b7-5d1e052ead91-cert\") pod \"odh-model-controller-75d9cdbddf-vcf8f\" (UID: \"b9369769-2bd9-4aba-b5b7-5d1e052ead91\") " pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:24.312769 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:24.312748    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d72388a7-93f1-42dc-aa06-2aac5e985452-tls-certs\") pod \"model-serving-api-7b954ffc95-qr44z\" (UID: \"d72388a7-93f1-42dc-aa06-2aac5e985452\") " pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:24.526385 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:24.526337    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:24.541978 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:24.541933    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:24.668062 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:24.668035    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-qr44z"]
Jul 28 13:02:24.670037 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:02:24.670000    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72388a7_93f1_42dc_aa06_2aac5e985452.slice/crio-6ced5aa23d93c2f3d9bf739e98124dae74af735dff0f22f9d72b06c22465608a WatchSource:0}: Error finding container 6ced5aa23d93c2f3d9bf739e98124dae74af735dff0f22f9d72b06c22465608a: Status 404 returned error can't find the container with id 6ced5aa23d93c2f3d9bf739e98124dae74af735dff0f22f9d72b06c22465608a
Jul 28 13:02:24.700173 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:24.700149    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-vcf8f"]
Jul 28 13:02:24.702272 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:02:24.702249    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9369769_2bd9_4aba_b5b7_5d1e052ead91.slice/crio-9b4381fb4c7dc184694a37a161662621a5e2fe2d50366def74c8da3fbd312998 WatchSource:0}: Error finding container 9b4381fb4c7dc184694a37a161662621a5e2fe2d50366def74c8da3fbd312998: Status 404 returned error can't find the container with id 9b4381fb4c7dc184694a37a161662621a5e2fe2d50366def74c8da3fbd312998
Jul 28 13:02:25.493759 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:25.493719    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-vcf8f" event={"ID":"b9369769-2bd9-4aba-b5b7-5d1e052ead91","Type":"ContainerStarted","Data":"9b4381fb4c7dc184694a37a161662621a5e2fe2d50366def74c8da3fbd312998"}
Jul 28 13:02:25.495167 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:25.495141    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-qr44z" event={"ID":"d72388a7-93f1-42dc-aa06-2aac5e985452","Type":"ContainerStarted","Data":"6ced5aa23d93c2f3d9bf739e98124dae74af735dff0f22f9d72b06c22465608a"}
Jul 28 13:02:29.523136 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:29.523093    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-qr44z" event={"ID":"d72388a7-93f1-42dc-aa06-2aac5e985452","Type":"ContainerStarted","Data":"639dabf686bc5be1d5d4b3eadf236bb4f09526e492932839df60e63e8c778106"}
Jul 28 13:02:29.523588 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:29.523283    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:02:29.524588 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:29.524566    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-vcf8f" event={"ID":"b9369769-2bd9-4aba-b5b7-5d1e052ead91","Type":"ContainerStarted","Data":"f4767aebd9081cb475cac377952ace2c7ee24fb44ba1f56ada93cad2e1115383"}
Jul 28 13:02:29.524708 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:29.524696    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:29.542278 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:29.542177    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-qr44z" podStartSLOduration=1.9248362810000001 podStartE2EDuration="6.542164287s" totalImagesPullingTime="4.617328006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:02:29.540896185 +0000 UTC m=+548.863889125" watchObservedRunningTime="2026-07-28 13:02:29.542164287 +0000 UTC m=+548.865157227"
Jul 28 13:02:29.556753 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:29.556704    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-vcf8f" podStartSLOduration=1.9156560059999999 podStartE2EDuration="6.556689621s" totalImagesPullingTime="4.641033615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:02:29.555223638 +0000 UTC m=+548.878216577" watchObservedRunningTime="2026-07-28 13:02:29.556689621 +0000 UTC m=+548.879682561"
Jul 28 13:02:40.531405 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:40.531374    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-vcf8f"
Jul 28 13:02:40.533393 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:02:40.533369    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-qr44z"
Jul 28 13:03:21.078881 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:03:21.078853    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:03:21.085377 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:03:21.085352    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:04:15.168324 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.168286    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"]
Jul 28 13:04:15.170470 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.170453    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.174519 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.174495    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:04:15.174681 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.174541    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-aee94-kube-rbac-proxy-sar-config\""
Jul 28 13:04:15.174681 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.174559    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:04:15.174681 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.174604    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-aee94-predictor-serving-cert\""
Jul 28 13:04:15.174681 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.174625    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pkltr\""
Jul 28 13:04:15.183541 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.183516    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"]
Jul 28 13:04:15.194045 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.194021    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-aee94-predictor-77f784b6f9-bztzp\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") " pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.194164 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.194076    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a7eb8e-a0fc-4ec9-934e-200262797f02-proxy-tls\") pod \"success-200-isvc-aee94-predictor-77f784b6f9-bztzp\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") " pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.194206 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.194176    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9svfv\" (UniqueName: \"kubernetes.io/projected/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-api-access-9svfv\") pod \"success-200-isvc-aee94-predictor-77f784b6f9-bztzp\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") " pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.295022 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.294986    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-aee94-predictor-77f784b6f9-bztzp\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") " pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.295250 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.295055    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a7eb8e-a0fc-4ec9-934e-200262797f02-proxy-tls\") pod \"success-200-isvc-aee94-predictor-77f784b6f9-bztzp\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") " pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.295250 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.295097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9svfv\" (UniqueName: \"kubernetes.io/projected/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-api-access-9svfv\") pod \"success-200-isvc-aee94-predictor-77f784b6f9-bztzp\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") " pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.296242 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.296197    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-aee94-predictor-77f784b6f9-bztzp\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") " pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.302406 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.302371    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a7eb8e-a0fc-4ec9-934e-200262797f02-proxy-tls\") pod \"success-200-isvc-aee94-predictor-77f784b6f9-bztzp\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") " pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.312276 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.312248    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9svfv\" (UniqueName: \"kubernetes.io/projected/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-api-access-9svfv\") pod \"success-200-isvc-aee94-predictor-77f784b6f9-bztzp\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") " pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.481343 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.481247    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:15.625044 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.624922    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"]
Jul 28 13:04:15.628778 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:04:15.628749    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a7eb8e_a0fc_4ec9_934e_200262797f02.slice/crio-00cf09220856c293364f4746f2438f909586684b4cd1efdf9e953c65e43b8be9 WatchSource:0}: Error finding container 00cf09220856c293364f4746f2438f909586684b4cd1efdf9e953c65e43b8be9: Status 404 returned error can't find the container with id 00cf09220856c293364f4746f2438f909586684b4cd1efdf9e953c65e43b8be9
Jul 28 13:04:15.978935 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:15.978895    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" event={"ID":"d9a7eb8e-a0fc-4ec9-934e-200262797f02","Type":"ContainerStarted","Data":"00cf09220856c293364f4746f2438f909586684b4cd1efdf9e953c65e43b8be9"}
Jul 28 13:04:28.049754 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:28.049717    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" event={"ID":"d9a7eb8e-a0fc-4ec9-934e-200262797f02","Type":"ContainerStarted","Data":"df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039"}
Jul 28 13:04:31.065721 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:31.065685    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" event={"ID":"d9a7eb8e-a0fc-4ec9-934e-200262797f02","Type":"ContainerStarted","Data":"b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff"}
Jul 28 13:04:31.066157 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:31.065814    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:31.085862 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:31.085813    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" podStartSLOduration=1.6772474640000001 podStartE2EDuration="16.085798233s" totalImagesPullingTime="14.408550769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:31.084592316 +0000 UTC m=+670.407585256" watchObservedRunningTime="2026-07-28 13:04:31.085798233 +0000 UTC m=+670.408791173"
Jul 28 13:04:32.070237 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:32.070203    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:32.071657 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:32.071625    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:04:33.074122 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:33.074080    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:04:38.079861 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:38.079827    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:04:38.080717 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:38.080688    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:04:48.081139 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:48.081095    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:04:58.081483 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:04:58.081440    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:05:08.080621 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:08.080574    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:05:18.082163 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:18.082070    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:05:49.549643 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.549604    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"]
Jul 28 13:05:49.550207 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.549865    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container" containerID="cri-o://df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039" gracePeriod=30
Jul 28 13:05:49.550207 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.549931    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kube-rbac-proxy" containerID="cri-o://b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff" gracePeriod=30
Jul 28 13:05:49.600594 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.600560    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"]
Jul 28 13:05:49.603984 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.603962    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.606030 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.605998    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c21f8-predictor-serving-cert\""
Jul 28 13:05:49.606030 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.606009    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c21f8-kube-rbac-proxy-sar-config\""
Jul 28 13:05:49.614587 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.614561    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"]
Jul 28 13:05:49.700753 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.700716    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqrx\" (UniqueName: \"kubernetes.io/projected/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-api-access-wxqrx\") pod \"success-200-isvc-c21f8-predictor-7856975d79-sfn98\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") " pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.700973 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.700801    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c21f8-predictor-7856975d79-sfn98\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") " pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.700973 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.700828    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-proxy-tls\") pod \"success-200-isvc-c21f8-predictor-7856975d79-sfn98\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") " pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.801817 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.801715    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqrx\" (UniqueName: \"kubernetes.io/projected/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-api-access-wxqrx\") pod \"success-200-isvc-c21f8-predictor-7856975d79-sfn98\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") " pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.801817 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.801786    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c21f8-predictor-7856975d79-sfn98\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") " pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.801817 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.801810    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-proxy-tls\") pod \"success-200-isvc-c21f8-predictor-7856975d79-sfn98\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") " pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.802741 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.802684    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c21f8-predictor-7856975d79-sfn98\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") " pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.804378 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.804357    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-proxy-tls\") pod \"success-200-isvc-c21f8-predictor-7856975d79-sfn98\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") " pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.809685 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.809658    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqrx\" (UniqueName: \"kubernetes.io/projected/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-api-access-wxqrx\") pod \"success-200-isvc-c21f8-predictor-7856975d79-sfn98\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") " pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:49.914805 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:49.914763    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:50.049001 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.048915    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"]
Jul 28 13:05:50.051551 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:05:50.051519    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e995b2_cbd3_49a9_bb2f_e1ad5ff6fddc.slice/crio-f493f1686305c02743ca173f5de70a631f9b74ca64d48f6fdb469759783b9790 WatchSource:0}: Error finding container f493f1686305c02743ca173f5de70a631f9b74ca64d48f6fdb469759783b9790: Status 404 returned error can't find the container with id f493f1686305c02743ca173f5de70a631f9b74ca64d48f6fdb469759783b9790
Jul 28 13:05:50.407184 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.407102    2572 generic.go:356] "Generic (PLEG): container finished" podID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerID="b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff" exitCode=2
Jul 28 13:05:50.407184 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.407172    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" event={"ID":"d9a7eb8e-a0fc-4ec9-934e-200262797f02","Type":"ContainerDied","Data":"b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff"}
Jul 28 13:05:50.410109 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.409387    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" event={"ID":"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc","Type":"ContainerStarted","Data":"538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364"}
Jul 28 13:05:50.410109 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.409455    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" event={"ID":"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc","Type":"ContainerStarted","Data":"03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7"}
Jul 28 13:05:50.410109 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.409484    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" event={"ID":"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc","Type":"ContainerStarted","Data":"f493f1686305c02743ca173f5de70a631f9b74ca64d48f6fdb469759783b9790"}
Jul 28 13:05:50.410502 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.410473    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:50.410603 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.410523    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:50.412340 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.412308    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 13:05:50.428182 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:50.428137    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podStartSLOduration=1.428122136 podStartE2EDuration="1.428122136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:05:50.427460719 +0000 UTC m=+749.750453681" watchObservedRunningTime="2026-07-28 13:05:50.428122136 +0000 UTC m=+749.751115117"
Jul 28 13:05:51.415304 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:51.415259    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 13:05:52.420065 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.420020    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 13:05:52.797471 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.797446    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:05:52.822118 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.822083    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9svfv\" (UniqueName: \"kubernetes.io/projected/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-api-access-9svfv\") pod \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") "
Jul 28 13:05:52.822290 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.822140    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a7eb8e-a0fc-4ec9-934e-200262797f02-proxy-tls\") pod \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") "
Jul 28 13:05:52.822290 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.822179    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-rbac-proxy-sar-config\") pod \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\" (UID: \"d9a7eb8e-a0fc-4ec9-934e-200262797f02\") "
Jul 28 13:05:52.822526 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.822499    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9a7eb8e-a0fc-4ec9-934e-200262797f02" (UID: "d9a7eb8e-a0fc-4ec9-934e-200262797f02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:05:52.824317 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.824292    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-api-access-9svfv" (OuterVolumeSpecName: "kube-api-access-9svfv") pod "d9a7eb8e-a0fc-4ec9-934e-200262797f02" (UID: "d9a7eb8e-a0fc-4ec9-934e-200262797f02"). InnerVolumeSpecName "kube-api-access-9svfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:05:52.824317 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.824304    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a7eb8e-a0fc-4ec9-934e-200262797f02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9a7eb8e-a0fc-4ec9-934e-200262797f02" (UID: "d9a7eb8e-a0fc-4ec9-934e-200262797f02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:05:52.923583 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.923543    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9svfv\" (UniqueName: \"kubernetes.io/projected/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-api-access-9svfv\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:05:52.923583 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.923572    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a7eb8e-a0fc-4ec9-934e-200262797f02-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:05:52.923583 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:52.923584    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a7eb8e-a0fc-4ec9-934e-200262797f02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:05:53.425572 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.425537    2572 generic.go:356] "Generic (PLEG): container finished" podID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerID="df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039" exitCode=0
Jul 28 13:05:53.426021 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.425590    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"
Jul 28 13:05:53.426021 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.425619    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" event={"ID":"d9a7eb8e-a0fc-4ec9-934e-200262797f02","Type":"ContainerDied","Data":"df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039"}
Jul 28 13:05:53.426021 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.425682    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp" event={"ID":"d9a7eb8e-a0fc-4ec9-934e-200262797f02","Type":"ContainerDied","Data":"00cf09220856c293364f4746f2438f909586684b4cd1efdf9e953c65e43b8be9"}
Jul 28 13:05:53.426021 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.425712    2572 scope.go:117] "RemoveContainer" containerID="b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff"
Jul 28 13:05:53.434165 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.434150    2572 scope.go:117] "RemoveContainer" containerID="df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039"
Jul 28 13:05:53.441382 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.441360    2572 scope.go:117] "RemoveContainer" containerID="b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff"
Jul 28 13:05:53.441622 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:05:53.441604    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff\": container with ID starting with b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff not found: ID does not exist" containerID="b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff"
Jul 28 13:05:53.441670 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.441632    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff"} err="failed to get container status \"b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff\": rpc error: code = NotFound desc = could not find container \"b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff\": container with ID starting with b8de63397bd08313f98202a23ced933268f9d225a1235ea2ac7e99fa93061eff not found: ID does not exist"
Jul 28 13:05:53.441670 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.441653    2572 scope.go:117] "RemoveContainer" containerID="df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039"
Jul 28 13:05:53.441892 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:05:53.441873    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039\": container with ID starting with df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039 not found: ID does not exist" containerID="df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039"
Jul 28 13:05:53.441930 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.441898    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039"} err="failed to get container status \"df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039\": rpc error: code = NotFound desc = could not find container \"df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039\": container with ID starting with df4f7cab0266dcf132eff48e7c106d97e30c274a127097eebf4bbd86e23ab039 not found: ID does not exist"
Jul 28 13:05:53.445771 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.445734    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"]
Jul 28 13:05:53.449847 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:53.449824    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-aee94-predictor-77f784b6f9-bztzp"]
Jul 28 13:05:55.180249 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:55.180219    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" path="/var/lib/kubelet/pods/d9a7eb8e-a0fc-4ec9-934e-200262797f02/volumes"
Jul 28 13:05:57.425058 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:57.425027    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:05:57.425611 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:05:57.425582    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 13:06:07.425743 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:07.425685    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 13:06:17.426178 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:17.426139    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 13:06:25.543810 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.543777    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"]
Jul 28 13:06:25.544207 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.544148    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kube-rbac-proxy"
Jul 28 13:06:25.544207 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.544160    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kube-rbac-proxy"
Jul 28 13:06:25.544207 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.544173    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container"
Jul 28 13:06:25.544207 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.544179    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container"
Jul 28 13:06:25.544374 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.544245    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kserve-container"
Jul 28 13:06:25.544374 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.544255    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a7eb8e-a0fc-4ec9-934e-200262797f02" containerName="kube-rbac-proxy"
Jul 28 13:06:25.548956 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.548924    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.551278 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.551256    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fec78-kube-rbac-proxy-sar-config\""
Jul 28 13:06:25.551445 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.551424    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fec78-predictor-serving-cert\""
Jul 28 13:06:25.559400 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.559379    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"]
Jul 28 13:06:25.699237 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.699198    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2r5b\" (UniqueName: \"kubernetes.io/projected/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-api-access-c2r5b\") pod \"success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") " pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.699436 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.699246    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-proxy-tls\") pod \"success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") " pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.699436 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.699351    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") " pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.800885 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.800800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") " pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.800885 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.800863    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2r5b\" (UniqueName: \"kubernetes.io/projected/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-api-access-c2r5b\") pod \"success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") " pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.801168 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.800888    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-proxy-tls\") pod \"success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") " pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.801552 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.801524    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") " pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.803407 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.803384    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-proxy-tls\") pod \"success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") " pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.818395 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.818357    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2r5b\" (UniqueName: \"kubernetes.io/projected/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-api-access-c2r5b\") pod \"success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") " pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:25.860186 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:25.860148    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:26.003970 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:26.003536    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"]
Jul 28 13:06:26.010583 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:06:26.010555    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01df65b5_2e77_4dcd_ac47_2a3d54165c5c.slice/crio-c5076cac8ff52928e2f7bed144528e6a9c36a152db957b6fd36ca47a673e30a9 WatchSource:0}: Error finding container c5076cac8ff52928e2f7bed144528e6a9c36a152db957b6fd36ca47a673e30a9: Status 404 returned error can't find the container with id c5076cac8ff52928e2f7bed144528e6a9c36a152db957b6fd36ca47a673e30a9
Jul 28 13:06:26.573034 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:26.572989    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" event={"ID":"01df65b5-2e77-4dcd-ac47-2a3d54165c5c","Type":"ContainerStarted","Data":"30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658"}
Jul 28 13:06:26.573449 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:26.573067    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" event={"ID":"01df65b5-2e77-4dcd-ac47-2a3d54165c5c","Type":"ContainerStarted","Data":"5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c"}
Jul 28 13:06:26.573449 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:26.573105    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:26.573449 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:26.573133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" event={"ID":"01df65b5-2e77-4dcd-ac47-2a3d54165c5c","Type":"ContainerStarted","Data":"c5076cac8ff52928e2f7bed144528e6a9c36a152db957b6fd36ca47a673e30a9"}
Jul 28 13:06:26.592147 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:26.592091    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podStartSLOduration=1.592075511 podStartE2EDuration="1.592075511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:06:26.59076713 +0000 UTC m=+785.913760071" watchObservedRunningTime="2026-07-28 13:06:26.592075511 +0000 UTC m=+785.915068450"
Jul 28 13:06:27.426518 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:27.426480    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 13:06:27.577733 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:27.577696    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:27.579177 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:27.579151    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 13:06:28.582453 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:28.582409    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 13:06:33.586953 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:33.586921    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:06:33.587534 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:33.587506    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 13:06:37.426651 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:37.426614    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:06:43.587925 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:43.587882    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 13:06:53.588105 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:06:53.588054    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 13:07:03.587703 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:07:03.587660    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 13:07:13.588958 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:07:13.588915    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:08:21.105767 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:08:21.105678    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:08:21.111785 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:08:21.111760    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:13:21.139715 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:13:21.139681    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:13:21.140657 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:13:21.140637    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:15:04.477351 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.477312    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"]
Jul 28 13:15:04.479904 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.477592    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container" containerID="cri-o://03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7" gracePeriod=30
Jul 28 13:15:04.479904 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.477640    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kube-rbac-proxy" containerID="cri-o://538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364" gracePeriod=30
Jul 28 13:15:04.605762 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.605726    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"]
Jul 28 13:15:04.609142 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.609121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:04.611425 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.611402    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dec07-kube-rbac-proxy-sar-config\""
Jul 28 13:15:04.611539 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.611402    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dec07-predictor-serving-cert\""
Jul 28 13:15:04.619813 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.619782    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"]
Jul 28 13:15:04.720307 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.720273    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l778q\" (UniqueName: \"kubernetes.io/projected/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-api-access-l778q\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:04.720489 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.720328    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:04.720489 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.720364    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:04.821605 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.821565    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l778q\" (UniqueName: \"kubernetes.io/projected/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-api-access-l778q\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:04.821802 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.821613    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:04.821802 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.821637    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:04.821931 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:15:04.821847    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-dec07-predictor-serving-cert: secret "success-200-isvc-dec07-predictor-serving-cert" not found
Jul 28 13:15:04.821931 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:15:04.821919    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls podName:6f82995e-c487-49fa-bf5e-ffe488d01982 nodeName:}" failed. No retries permitted until 2026-07-28 13:15:05.321897422 +0000 UTC m=+1304.644890341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls") pod "success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" (UID: "6f82995e-c487-49fa-bf5e-ffe488d01982") : secret "success-200-isvc-dec07-predictor-serving-cert" not found
Jul 28 13:15:04.822436 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.822412    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:04.830291 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.830258    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l778q\" (UniqueName: \"kubernetes.io/projected/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-api-access-l778q\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:04.842257 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.842224    2572 generic.go:356] "Generic (PLEG): container finished" podID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerID="538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364" exitCode=2
Jul 28 13:15:04.842398 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:04.842299    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" event={"ID":"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc","Type":"ContainerDied","Data":"538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364"}
Jul 28 13:15:05.326052 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:05.326012    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:05.326236 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:15:05.326170    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-dec07-predictor-serving-cert: secret "success-200-isvc-dec07-predictor-serving-cert" not found
Jul 28 13:15:05.326277 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:15:05.326236    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls podName:6f82995e-c487-49fa-bf5e-ffe488d01982 nodeName:}" failed. No retries permitted until 2026-07-28 13:15:06.326217257 +0000 UTC m=+1305.649210179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls") pod "success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" (UID: "6f82995e-c487-49fa-bf5e-ffe488d01982") : secret "success-200-isvc-dec07-predictor-serving-cert" not found
Jul 28 13:15:06.334803 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.334761    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:06.337222 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.337202    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls\") pod \"success-200-isvc-dec07-predictor-6dcc759d57-v8lbc\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") " pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:06.421683 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.421645    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:06.552841 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.552800    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"]
Jul 28 13:15:06.557470 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:15:06.557446    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f82995e_c487_49fa_bf5e_ffe488d01982.slice/crio-2734fd78f101500aad25fa754bfafbf6d34eefdb500a9d9f2cfafb6d7eec3829 WatchSource:0}: Error finding container 2734fd78f101500aad25fa754bfafbf6d34eefdb500a9d9f2cfafb6d7eec3829: Status 404 returned error can't find the container with id 2734fd78f101500aad25fa754bfafbf6d34eefdb500a9d9f2cfafb6d7eec3829
Jul 28 13:15:06.854214 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.854120    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" event={"ID":"6f82995e-c487-49fa-bf5e-ffe488d01982","Type":"ContainerStarted","Data":"11cef274a0fd616e9fe960e2737b5110ee09a31f5d5eb2e3727e0592204beda6"}
Jul 28 13:15:06.854214 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.854178    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" event={"ID":"6f82995e-c487-49fa-bf5e-ffe488d01982","Type":"ContainerStarted","Data":"cd3944022812285e74ca4ed1056cee8a1a64145f5b649f801892bb0d5867400e"}
Jul 28 13:15:06.854214 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.854199    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" event={"ID":"6f82995e-c487-49fa-bf5e-ffe488d01982","Type":"ContainerStarted","Data":"2734fd78f101500aad25fa754bfafbf6d34eefdb500a9d9f2cfafb6d7eec3829"}
Jul 28 13:15:06.854492 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.854221    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:06.854492 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.854240    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:06.855891 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.855863    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:15:06.873072 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:06.873024    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podStartSLOduration=2.873008941 podStartE2EDuration="2.873008941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:15:06.871823762 +0000 UTC m=+1306.194816708" watchObservedRunningTime="2026-07-28 13:15:06.873008941 +0000 UTC m=+1306.196001881"
Jul 28 13:15:07.421289 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:07.421246    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 13:15:07.425648 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:07.425616    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 13:15:07.859000 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:07.858962    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:15:08.138998 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.138971    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:15:08.251831 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.251799    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-rbac-proxy-sar-config\") pod \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") "
Jul 28 13:15:08.251831 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.251844    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxqrx\" (UniqueName: \"kubernetes.io/projected/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-api-access-wxqrx\") pod \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") "
Jul 28 13:15:08.252126 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.251886    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-proxy-tls\") pod \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\" (UID: \"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc\") "
Jul 28 13:15:08.252302 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.252266    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" (UID: "e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:15:08.254085 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.254058    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" (UID: "e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:15:08.254181 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.254095    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-api-access-wxqrx" (OuterVolumeSpecName: "kube-api-access-wxqrx") pod "e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" (UID: "e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc"). InnerVolumeSpecName "kube-api-access-wxqrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:15:08.353000 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.352936    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:15:08.353000 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.352993    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:15:08.353000 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.353003    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxqrx\" (UniqueName: \"kubernetes.io/projected/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc-kube-api-access-wxqrx\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:15:08.864534 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.864493    2572 generic.go:356] "Generic (PLEG): container finished" podID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerID="03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7" exitCode=0
Jul 28 13:15:08.864930 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.864546    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"
Jul 28 13:15:08.864930 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.864573    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" event={"ID":"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc","Type":"ContainerDied","Data":"03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7"}
Jul 28 13:15:08.864930 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.864641    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98" event={"ID":"e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc","Type":"ContainerDied","Data":"f493f1686305c02743ca173f5de70a631f9b74ca64d48f6fdb469759783b9790"}
Jul 28 13:15:08.864930 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.864668    2572 scope.go:117] "RemoveContainer" containerID="538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364"
Jul 28 13:15:08.874307 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.874289    2572 scope.go:117] "RemoveContainer" containerID="03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7"
Jul 28 13:15:08.882068 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.882047    2572 scope.go:117] "RemoveContainer" containerID="538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364"
Jul 28 13:15:08.882375 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:15:08.882354    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364\": container with ID starting with 538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364 not found: ID does not exist" containerID="538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364"
Jul 28 13:15:08.882414 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.882386    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364"} err="failed to get container status \"538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364\": rpc error: code = NotFound desc = could not find container \"538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364\": container with ID starting with 538c02e77e89741203550167ec18d070d241cb03ae4a66b611ef74cf48704364 not found: ID does not exist"
Jul 28 13:15:08.882414 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.882406    2572 scope.go:117] "RemoveContainer" containerID="03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7"
Jul 28 13:15:08.882665 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:15:08.882632    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7\": container with ID starting with 03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7 not found: ID does not exist" containerID="03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7"
Jul 28 13:15:08.882701 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.882673    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7"} err="failed to get container status \"03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7\": rpc error: code = NotFound desc = could not find container \"03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7\": container with ID starting with 03e0d30d3d4801ac000fac0b42c00471867649b399cda46a5c09f7ce6c7eaaf7 not found: ID does not exist"
Jul 28 13:15:08.885526 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.885501    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"]
Jul 28 13:15:08.889468 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:08.889447    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c21f8-predictor-7856975d79-sfn98"]
Jul 28 13:15:09.180969 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:09.180882    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" path="/var/lib/kubelet/pods/e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc/volumes"
Jul 28 13:15:12.863430 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:12.863398    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:12.864035 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:12.864007    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:15:22.864627 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:22.864583    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:15:32.864955 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:32.864904    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:15:40.344870 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.344835    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"]
Jul 28 13:15:40.345387 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.345163    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container" containerID="cri-o://5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c" gracePeriod=30
Jul 28 13:15:40.345387 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.345215    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kube-rbac-proxy" containerID="cri-o://30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658" gracePeriod=30
Jul 28 13:15:40.398417 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.398381    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"]
Jul 28 13:15:40.398834 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.398806    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kube-rbac-proxy"
Jul 28 13:15:40.398834 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.398830    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kube-rbac-proxy"
Jul 28 13:15:40.399039 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.398844    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container"
Jul 28 13:15:40.399039 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.398852    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container"
Jul 28 13:15:40.399039 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.398924    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kserve-container"
Jul 28 13:15:40.399039 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.398955    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e995b2-cbd3-49a9-bb2f-e1ad5ff6fddc" containerName="kube-rbac-proxy"
Jul 28 13:15:40.403405 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.403381    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.405387 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.405366    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7ff2e-predictor-serving-cert\""
Jul 28 13:15:40.405553 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.405535    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7ff2e-kube-rbac-proxy-sar-config\""
Jul 28 13:15:40.413896 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.413862    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"]
Jul 28 13:15:40.524753 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.524719    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31815b80-5909-48c4-b48f-1060343f97be-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") " pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.524930 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.524772    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9glm\" (UniqueName: \"kubernetes.io/projected/31815b80-5909-48c4-b48f-1060343f97be-kube-api-access-s9glm\") pod \"success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") " pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.524930 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.524835    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31815b80-5909-48c4-b48f-1060343f97be-proxy-tls\") pod \"success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") " pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.625695 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.625612    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31815b80-5909-48c4-b48f-1060343f97be-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") " pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.625695 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.625662    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9glm\" (UniqueName: \"kubernetes.io/projected/31815b80-5909-48c4-b48f-1060343f97be-kube-api-access-s9glm\") pod \"success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") " pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.625904 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.625702    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31815b80-5909-48c4-b48f-1060343f97be-proxy-tls\") pod \"success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") " pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.626339 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.626311    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31815b80-5909-48c4-b48f-1060343f97be-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") " pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.628203 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.628182    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31815b80-5909-48c4-b48f-1060343f97be-proxy-tls\") pod \"success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") " pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.633523 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.633499    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9glm\" (UniqueName: \"kubernetes.io/projected/31815b80-5909-48c4-b48f-1060343f97be-kube-api-access-s9glm\") pod \"success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") " pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.716557 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.716509    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:40.856870 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:40.856838    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"]
Jul 28 13:15:40.859785 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:15:40.859755    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31815b80_5909_48c4_b48f_1060343f97be.slice/crio-7a74c7c400055e4c0fc07aa272df8fc8390731ae517d1fd81f56013230b48907 WatchSource:0}: Error finding container 7a74c7c400055e4c0fc07aa272df8fc8390731ae517d1fd81f56013230b48907: Status 404 returned error can't find the container with id 7a74c7c400055e4c0fc07aa272df8fc8390731ae517d1fd81f56013230b48907
Jul 28 13:15:41.003605 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:41.003573    2572 generic.go:356] "Generic (PLEG): container finished" podID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerID="30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658" exitCode=2
Jul 28 13:15:41.003777 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:41.003652    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" event={"ID":"01df65b5-2e77-4dcd-ac47-2a3d54165c5c","Type":"ContainerDied","Data":"30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658"}
Jul 28 13:15:41.005361 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:41.005332    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" event={"ID":"31815b80-5909-48c4-b48f-1060343f97be","Type":"ContainerStarted","Data":"c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f"}
Jul 28 13:15:41.005491 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:41.005385    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:41.005491 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:41.005411    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" event={"ID":"31815b80-5909-48c4-b48f-1060343f97be","Type":"ContainerStarted","Data":"443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46"}
Jul 28 13:15:41.005491 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:41.005430    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" event={"ID":"31815b80-5909-48c4-b48f-1060343f97be","Type":"ContainerStarted","Data":"7a74c7c400055e4c0fc07aa272df8fc8390731ae517d1fd81f56013230b48907"}
Jul 28 13:15:41.005635 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:41.005534    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:41.007090 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:41.007063    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 13:15:41.022986 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:41.022868    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podStartSLOduration=1.022848792 podStartE2EDuration="1.022848792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:15:41.022358696 +0000 UTC m=+1340.345351663" watchObservedRunningTime="2026-07-28 13:15:41.022848792 +0000 UTC m=+1340.345841734"
Jul 28 13:15:42.011015 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:42.010972    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 13:15:42.864392 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:42.864355    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:15:43.605633 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.605610    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:15:43.753810 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.753666    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-rbac-proxy-sar-config\") pod \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") "
Jul 28 13:15:43.753810 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.753737    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2r5b\" (UniqueName: \"kubernetes.io/projected/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-api-access-c2r5b\") pod \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") "
Jul 28 13:15:43.753810 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.753787    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-proxy-tls\") pod \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\" (UID: \"01df65b5-2e77-4dcd-ac47-2a3d54165c5c\") "
Jul 28 13:15:43.754205 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.754178    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01df65b5-2e77-4dcd-ac47-2a3d54165c5c" (UID: "01df65b5-2e77-4dcd-ac47-2a3d54165c5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:15:43.755968 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.755915    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-api-access-c2r5b" (OuterVolumeSpecName: "kube-api-access-c2r5b") pod "01df65b5-2e77-4dcd-ac47-2a3d54165c5c" (UID: "01df65b5-2e77-4dcd-ac47-2a3d54165c5c"). InnerVolumeSpecName "kube-api-access-c2r5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:15:43.756095 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.756021    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01df65b5-2e77-4dcd-ac47-2a3d54165c5c" (UID: "01df65b5-2e77-4dcd-ac47-2a3d54165c5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:15:43.854840 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.854802    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2r5b\" (UniqueName: \"kubernetes.io/projected/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-api-access-c2r5b\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:15:43.854840 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.854835    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:15:43.854840 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:43.854845    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01df65b5-2e77-4dcd-ac47-2a3d54165c5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:15:44.021408 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.021373    2572 generic.go:356] "Generic (PLEG): container finished" podID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerID="5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c" exitCode=0
Jul 28 13:15:44.021557 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.021423    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"
Jul 28 13:15:44.021557 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.021462    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" event={"ID":"01df65b5-2e77-4dcd-ac47-2a3d54165c5c","Type":"ContainerDied","Data":"5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c"}
Jul 28 13:15:44.021557 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.021522    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" event={"ID":"01df65b5-2e77-4dcd-ac47-2a3d54165c5c","Type":"ContainerDied","Data":"c5076cac8ff52928e2f7bed144528e6a9c36a152db957b6fd36ca47a673e30a9"}
Jul 28 13:15:44.021557 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.021549    2572 scope.go:117] "RemoveContainer" containerID="30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658"
Jul 28 13:15:44.030930 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.030914    2572 scope.go:117] "RemoveContainer" containerID="5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c"
Jul 28 13:15:44.038344 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.038326    2572 scope.go:117] "RemoveContainer" containerID="30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658"
Jul 28 13:15:44.038589 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:15:44.038569    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658\": container with ID starting with 30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658 not found: ID does not exist" containerID="30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658"
Jul 28 13:15:44.038639 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.038598    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658"} err="failed to get container status \"30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658\": rpc error: code = NotFound desc = could not find container \"30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658\": container with ID starting with 30447d50d92639fb8bbd1fd84d7ab163de2defa1927b2c3b85dca6b1b35f4658 not found: ID does not exist"
Jul 28 13:15:44.038639 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.038614    2572 scope.go:117] "RemoveContainer" containerID="5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c"
Jul 28 13:15:44.038840 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:15:44.038822    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c\": container with ID starting with 5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c not found: ID does not exist" containerID="5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c"
Jul 28 13:15:44.038883 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.038847    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c"} err="failed to get container status \"5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c\": rpc error: code = NotFound desc = could not find container \"5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c\": container with ID starting with 5f85a0dabed47e006f1d943f1f5c003862c6b00f158160ba116ee2a7d37f157c not found: ID does not exist"
Jul 28 13:15:44.042474 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.042451    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"]
Jul 28 13:15:44.046066 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.046045    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6"]
Jul 28 13:15:44.583927 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.583883    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 13:15:44.588031 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:44.588004    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fec78-predictor-55f58fdf5c-5lvt6" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: i/o timeout"
Jul 28 13:15:45.181743 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:45.181706    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" path="/var/lib/kubelet/pods/01df65b5-2e77-4dcd-ac47-2a3d54165c5c/volumes"
Jul 28 13:15:47.016432 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:47.016405    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:15:47.017043 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:47.016910    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 13:15:52.865798 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:52.865762    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:15:57.017608 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:15:57.017565    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 13:16:07.017194 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:07.017154    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 13:16:14.790796 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.790756    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"]
Jul 28 13:16:14.791267 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.791073    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container" containerID="cri-o://cd3944022812285e74ca4ed1056cee8a1a64145f5b649f801892bb0d5867400e" gracePeriod=30
Jul 28 13:16:14.791267 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.791108    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kube-rbac-proxy" containerID="cri-o://11cef274a0fd616e9fe960e2737b5110ee09a31f5d5eb2e3727e0592204beda6" gracePeriod=30
Jul 28 13:16:14.870092 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.870056    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"]
Jul 28 13:16:14.870414 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.870403    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container"
Jul 28 13:16:14.870465 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.870417    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container"
Jul 28 13:16:14.870465 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.870426    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kube-rbac-proxy"
Jul 28 13:16:14.870465 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.870431    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kube-rbac-proxy"
Jul 28 13:16:14.870556 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.870493    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kube-rbac-proxy"
Jul 28 13:16:14.870556 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.870502    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="01df65b5-2e77-4dcd-ac47-2a3d54165c5c" containerName="kserve-container"
Jul 28 13:16:14.873867 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.873849    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:14.876080 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.876055    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7cae9-predictor-serving-cert\""
Jul 28 13:16:14.876191 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.876080    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7cae9-kube-rbac-proxy-sar-config\""
Jul 28 13:16:14.882383 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.882356    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"]
Jul 28 13:16:14.908201 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.908172    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qggx6\" (UniqueName: \"kubernetes.io/projected/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-api-access-qggx6\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:14.908351 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.908210    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:14.908351 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:14.908314    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0da8a2-05d2-4e64-ae26-3c6478791e82-proxy-tls\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:15.009401 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.009357    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0da8a2-05d2-4e64-ae26-3c6478791e82-proxy-tls\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:15.009616 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.009446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qggx6\" (UniqueName: \"kubernetes.io/projected/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-api-access-qggx6\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:15.009616 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.009482    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:15.009616 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:16:15.009525    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-serving-cert: secret "success-200-isvc-7cae9-predictor-serving-cert" not found
Jul 28 13:16:15.009616 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:16:15.009595    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0da8a2-05d2-4e64-ae26-3c6478791e82-proxy-tls podName:7b0da8a2-05d2-4e64-ae26-3c6478791e82 nodeName:}" failed. No retries permitted until 2026-07-28 13:16:15.50957223 +0000 UTC m=+1374.832565165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7b0da8a2-05d2-4e64-ae26-3c6478791e82-proxy-tls") pod "success-200-isvc-7cae9-predictor-7fc49f875c-n572r" (UID: "7b0da8a2-05d2-4e64-ae26-3c6478791e82") : secret "success-200-isvc-7cae9-predictor-serving-cert" not found
Jul 28 13:16:15.010233 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.010212    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:15.017933 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.017901    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qggx6\" (UniqueName: \"kubernetes.io/projected/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-api-access-qggx6\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:15.163284 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.163191    2572 generic.go:356] "Generic (PLEG): container finished" podID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerID="11cef274a0fd616e9fe960e2737b5110ee09a31f5d5eb2e3727e0592204beda6" exitCode=2
Jul 28 13:16:15.163284 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.163252    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" event={"ID":"6f82995e-c487-49fa-bf5e-ffe488d01982","Type":"ContainerDied","Data":"11cef274a0fd616e9fe960e2737b5110ee09a31f5d5eb2e3727e0592204beda6"}
Jul 28 13:16:15.513957 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.513892    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0da8a2-05d2-4e64-ae26-3c6478791e82-proxy-tls\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:15.516338 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.516311    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0da8a2-05d2-4e64-ae26-3c6478791e82-proxy-tls\") pod \"success-200-isvc-7cae9-predictor-7fc49f875c-n572r\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") " pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:15.784842 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.784747    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:15.921739 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:15.921685    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"]
Jul 28 13:16:15.924126 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:16:15.924097    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0da8a2_05d2_4e64_ae26_3c6478791e82.slice/crio-54b74ef34928b6c463d3ca1bd478e714f69bb19b540c56eef060a168cfe51761 WatchSource:0}: Error finding container 54b74ef34928b6c463d3ca1bd478e714f69bb19b540c56eef060a168cfe51761: Status 404 returned error can't find the container with id 54b74ef34928b6c463d3ca1bd478e714f69bb19b540c56eef060a168cfe51761
Jul 28 13:16:16.169230 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:16.169196    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" event={"ID":"7b0da8a2-05d2-4e64-ae26-3c6478791e82","Type":"ContainerStarted","Data":"11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda"}
Jul 28 13:16:16.169425 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:16.169255    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" event={"ID":"7b0da8a2-05d2-4e64-ae26-3c6478791e82","Type":"ContainerStarted","Data":"7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379"}
Jul 28 13:16:16.169472 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:16.169422    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:16.169524 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:16.169470    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" event={"ID":"7b0da8a2-05d2-4e64-ae26-3c6478791e82","Type":"ContainerStarted","Data":"54b74ef34928b6c463d3ca1bd478e714f69bb19b540c56eef060a168cfe51761"}
Jul 28 13:16:16.169524 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:16.169503    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:16.171255 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:16.171228    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 13:16:16.186715 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:16.186672    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podStartSLOduration=2.186657152 podStartE2EDuration="2.186657152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:16:16.184822271 +0000 UTC m=+1375.507815211" watchObservedRunningTime="2026-07-28 13:16:16.186657152 +0000 UTC m=+1375.509650092"
Jul 28 13:16:17.017884 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:17.017831    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 13:16:17.173812 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:17.173775    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 13:16:17.860026 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:17.859985    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 13:16:18.180665 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.180630    2572 generic.go:356] "Generic (PLEG): container finished" podID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerID="cd3944022812285e74ca4ed1056cee8a1a64145f5b649f801892bb0d5867400e" exitCode=0
Jul 28 13:16:18.181162 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.180712    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" event={"ID":"6f82995e-c487-49fa-bf5e-ffe488d01982","Type":"ContainerDied","Data":"cd3944022812285e74ca4ed1056cee8a1a64145f5b649f801892bb0d5867400e"}
Jul 28 13:16:18.250800 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.250775    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:16:18.343268 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.343234    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls\") pod \"6f82995e-c487-49fa-bf5e-ffe488d01982\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") "
Jul 28 13:16:18.343268 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.343284    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l778q\" (UniqueName: \"kubernetes.io/projected/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-api-access-l778q\") pod \"6f82995e-c487-49fa-bf5e-ffe488d01982\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") "
Jul 28 13:16:18.343480 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.343312    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-rbac-proxy-sar-config\") pod \"6f82995e-c487-49fa-bf5e-ffe488d01982\" (UID: \"6f82995e-c487-49fa-bf5e-ffe488d01982\") "
Jul 28 13:16:18.343702 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.343677    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f82995e-c487-49fa-bf5e-ffe488d01982" (UID: "6f82995e-c487-49fa-bf5e-ffe488d01982"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:16:18.345389 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.345366    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f82995e-c487-49fa-bf5e-ffe488d01982" (UID: "6f82995e-c487-49fa-bf5e-ffe488d01982"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:16:18.345525 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.345501    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-api-access-l778q" (OuterVolumeSpecName: "kube-api-access-l778q") pod "6f82995e-c487-49fa-bf5e-ffe488d01982" (UID: "6f82995e-c487-49fa-bf5e-ffe488d01982"). InnerVolumeSpecName "kube-api-access-l778q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:16:18.444855 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.444760    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f82995e-c487-49fa-bf5e-ffe488d01982-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:16:18.444855 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.444795    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l778q\" (UniqueName: \"kubernetes.io/projected/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-api-access-l778q\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:16:18.444855 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:18.444805    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f82995e-c487-49fa-bf5e-ffe488d01982-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:16:19.187139 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:19.187105    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"
Jul 28 13:16:19.187613 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:19.187108    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc" event={"ID":"6f82995e-c487-49fa-bf5e-ffe488d01982","Type":"ContainerDied","Data":"2734fd78f101500aad25fa754bfafbf6d34eefdb500a9d9f2cfafb6d7eec3829"}
Jul 28 13:16:19.187613 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:19.187261    2572 scope.go:117] "RemoveContainer" containerID="11cef274a0fd616e9fe960e2737b5110ee09a31f5d5eb2e3727e0592204beda6"
Jul 28 13:16:19.197809 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:19.197782    2572 scope.go:117] "RemoveContainer" containerID="cd3944022812285e74ca4ed1056cee8a1a64145f5b649f801892bb0d5867400e"
Jul 28 13:16:19.207062 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:19.207035    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"]
Jul 28 13:16:19.210452 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:19.210422    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dec07-predictor-6dcc759d57-v8lbc"]
Jul 28 13:16:21.180876 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:21.180842    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" path="/var/lib/kubelet/pods/6f82995e-c487-49fa-bf5e-ffe488d01982/volumes"
Jul 28 13:16:22.179115 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:22.179087    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:16:22.179540 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:22.179509    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 13:16:27.017788 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:27.017749    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:16:32.179529 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:32.179484    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 13:16:42.180142 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:42.180102    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 13:16:50.614560 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.614524    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"]
Jul 28 13:16:50.615183 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.614811    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container" containerID="cri-o://443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46" gracePeriod=30
Jul 28 13:16:50.615183 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.614883    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kube-rbac-proxy" containerID="cri-o://c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f" gracePeriod=30
Jul 28 13:16:50.684650 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.684620    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"]
Jul 28 13:16:50.684991 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.684974    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container"
Jul 28 13:16:50.685045 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.684995    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container"
Jul 28 13:16:50.685045 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.685022    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kube-rbac-proxy"
Jul 28 13:16:50.685045 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.685031    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kube-rbac-proxy"
Jul 28 13:16:50.685150 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.685105    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kserve-container"
Jul 28 13:16:50.685150 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.685117    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f82995e-c487-49fa-bf5e-ffe488d01982" containerName="kube-rbac-proxy"
Jul 28 13:16:50.689704 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.689684    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:50.691745 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.691727    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dd548-predictor-serving-cert\""
Jul 28 13:16:50.691831 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.691750    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dd548-kube-rbac-proxy-sar-config\""
Jul 28 13:16:50.698041 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.698014    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"]
Jul 28 13:16:50.820407 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.820373    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6092f68a-0296-4036-ae75-020b8de66fc4-proxy-tls\") pod \"success-200-isvc-dd548-predictor-d86d86669-w2r2p\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") " pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:50.820583 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.820427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6092f68a-0296-4036-ae75-020b8de66fc4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dd548-predictor-d86d86669-w2r2p\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") " pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:50.820583 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.820485    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7ng\" (UniqueName: \"kubernetes.io/projected/6092f68a-0296-4036-ae75-020b8de66fc4-kube-api-access-9q7ng\") pod \"success-200-isvc-dd548-predictor-d86d86669-w2r2p\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") " pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:50.921790 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.921712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6092f68a-0296-4036-ae75-020b8de66fc4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dd548-predictor-d86d86669-w2r2p\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") " pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:50.921790 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.921763    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7ng\" (UniqueName: \"kubernetes.io/projected/6092f68a-0296-4036-ae75-020b8de66fc4-kube-api-access-9q7ng\") pod \"success-200-isvc-dd548-predictor-d86d86669-w2r2p\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") " pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:50.922052 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.921822    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6092f68a-0296-4036-ae75-020b8de66fc4-proxy-tls\") pod \"success-200-isvc-dd548-predictor-d86d86669-w2r2p\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") " pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:50.922480 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.922453    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6092f68a-0296-4036-ae75-020b8de66fc4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dd548-predictor-d86d86669-w2r2p\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") " pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:50.924328 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.924307    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6092f68a-0296-4036-ae75-020b8de66fc4-proxy-tls\") pod \"success-200-isvc-dd548-predictor-d86d86669-w2r2p\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") " pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:50.928956 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:50.928923    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7ng\" (UniqueName: \"kubernetes.io/projected/6092f68a-0296-4036-ae75-020b8de66fc4-kube-api-access-9q7ng\") pod \"success-200-isvc-dd548-predictor-d86d86669-w2r2p\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") " pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:51.000874 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.000834    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:51.134481 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.134450    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"]
Jul 28 13:16:51.136985 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:16:51.136954    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6092f68a_0296_4036_ae75_020b8de66fc4.slice/crio-0f26cd3bd0c73168d16e3c5bd7a0bdcf02884d30746beda3c341468e62f342b6 WatchSource:0}: Error finding container 0f26cd3bd0c73168d16e3c5bd7a0bdcf02884d30746beda3c341468e62f342b6: Status 404 returned error can't find the container with id 0f26cd3bd0c73168d16e3c5bd7a0bdcf02884d30746beda3c341468e62f342b6
Jul 28 13:16:51.334329 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.334283    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" event={"ID":"6092f68a-0296-4036-ae75-020b8de66fc4","Type":"ContainerStarted","Data":"9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b"}
Jul 28 13:16:51.334518 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.334361    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:51.334518 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.334396    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" event={"ID":"6092f68a-0296-4036-ae75-020b8de66fc4","Type":"ContainerStarted","Data":"55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8"}
Jul 28 13:16:51.334518 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.334432    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:51.334518 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.334460    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" event={"ID":"6092f68a-0296-4036-ae75-020b8de66fc4","Type":"ContainerStarted","Data":"0f26cd3bd0c73168d16e3c5bd7a0bdcf02884d30746beda3c341468e62f342b6"}
Jul 28 13:16:51.336166 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.336136    2572 generic.go:356] "Generic (PLEG): container finished" podID="31815b80-5909-48c4-b48f-1060343f97be" containerID="c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f" exitCode=2
Jul 28 13:16:51.336276 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.336172    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" event={"ID":"31815b80-5909-48c4-b48f-1060343f97be","Type":"ContainerDied","Data":"c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f"}
Jul 28 13:16:51.336442 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.336419    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 13:16:51.354450 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:51.354401    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podStartSLOduration=1.35438825 podStartE2EDuration="1.35438825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:16:51.353551074 +0000 UTC m=+1410.676544006" watchObservedRunningTime="2026-07-28 13:16:51.35438825 +0000 UTC m=+1410.677381191"
Jul 28 13:16:52.012177 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:52.012125    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 28 13:16:52.180392 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:52.180352    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 13:16:52.340292 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:52.340202    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 13:16:53.874455 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:53.874432    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:16:53.945964 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:53.945910    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31815b80-5909-48c4-b48f-1060343f97be-proxy-tls\") pod \"31815b80-5909-48c4-b48f-1060343f97be\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") "
Jul 28 13:16:53.946156 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:53.946008    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31815b80-5909-48c4-b48f-1060343f97be-kube-rbac-proxy-sar-config\") pod \"31815b80-5909-48c4-b48f-1060343f97be\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") "
Jul 28 13:16:53.946156 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:53.946032    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9glm\" (UniqueName: \"kubernetes.io/projected/31815b80-5909-48c4-b48f-1060343f97be-kube-api-access-s9glm\") pod \"31815b80-5909-48c4-b48f-1060343f97be\" (UID: \"31815b80-5909-48c4-b48f-1060343f97be\") "
Jul 28 13:16:53.946413 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:53.946386    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31815b80-5909-48c4-b48f-1060343f97be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31815b80-5909-48c4-b48f-1060343f97be" (UID: "31815b80-5909-48c4-b48f-1060343f97be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:16:53.948151 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:53.948128    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31815b80-5909-48c4-b48f-1060343f97be-kube-api-access-s9glm" (OuterVolumeSpecName: "kube-api-access-s9glm") pod "31815b80-5909-48c4-b48f-1060343f97be" (UID: "31815b80-5909-48c4-b48f-1060343f97be"). InnerVolumeSpecName "kube-api-access-s9glm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:16:53.948216 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:53.948192    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31815b80-5909-48c4-b48f-1060343f97be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31815b80-5909-48c4-b48f-1060343f97be" (UID: "31815b80-5909-48c4-b48f-1060343f97be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:16:54.046832 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.046790    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31815b80-5909-48c4-b48f-1060343f97be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:16:54.046832 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.046824    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9glm\" (UniqueName: \"kubernetes.io/projected/31815b80-5909-48c4-b48f-1060343f97be-kube-api-access-s9glm\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:16:54.046832 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.046837    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31815b80-5909-48c4-b48f-1060343f97be-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:16:54.350519 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.350426    2572 generic.go:356] "Generic (PLEG): container finished" podID="31815b80-5909-48c4-b48f-1060343f97be" containerID="443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46" exitCode=0
Jul 28 13:16:54.350519 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.350480    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"
Jul 28 13:16:54.350519 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.350506    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" event={"ID":"31815b80-5909-48c4-b48f-1060343f97be","Type":"ContainerDied","Data":"443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46"}
Jul 28 13:16:54.350741 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.350561    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22" event={"ID":"31815b80-5909-48c4-b48f-1060343f97be","Type":"ContainerDied","Data":"7a74c7c400055e4c0fc07aa272df8fc8390731ae517d1fd81f56013230b48907"}
Jul 28 13:16:54.350741 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.350587    2572 scope.go:117] "RemoveContainer" containerID="c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f"
Jul 28 13:16:54.359458 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.359441    2572 scope.go:117] "RemoveContainer" containerID="443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46"
Jul 28 13:16:54.367386 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.367362    2572 scope.go:117] "RemoveContainer" containerID="c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f"
Jul 28 13:16:54.367640 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:16:54.367624    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f\": container with ID starting with c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f not found: ID does not exist" containerID="c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f"
Jul 28 13:16:54.367692 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.367649    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f"} err="failed to get container status \"c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f\": rpc error: code = NotFound desc = could not find container \"c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f\": container with ID starting with c9dc43f78ea32fe77af62a0811b46c39b3b8056811e2a840d5af8859d18ad10f not found: ID does not exist"
Jul 28 13:16:54.367692 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.367667    2572 scope.go:117] "RemoveContainer" containerID="443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46"
Jul 28 13:16:54.367927 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:16:54.367907    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46\": container with ID starting with 443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46 not found: ID does not exist" containerID="443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46"
Jul 28 13:16:54.367991 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.367933    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46"} err="failed to get container status \"443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46\": rpc error: code = NotFound desc = could not find container \"443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46\": container with ID starting with 443d3a57f6409a234f9a22261802e607a99c0bc017392b5b8a0435c50702de46 not found: ID does not exist"
Jul 28 13:16:54.373865 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.373841    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"]
Jul 28 13:16:54.377202 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:54.377182    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7ff2e-predictor-7cfd8b8f6c-d8t22"]
Jul 28 13:16:55.180430 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:55.180396    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31815b80-5909-48c4-b48f-1060343f97be" path="/var/lib/kubelet/pods/31815b80-5909-48c4-b48f-1060343f97be/volumes"
Jul 28 13:16:57.344810 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:57.344782    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:16:57.345234 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:16:57.345211    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 13:17:02.181091 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:17:02.181064    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:17:07.345447 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:17:07.345405    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 13:17:17.345435 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:17:17.345386    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 13:17:27.345373 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:17:27.345330    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 13:17:37.346360 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:17:37.346327    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:18:21.166554 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:18:21.166525    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:18:21.167716 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:18:21.167696    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:23:21.193123 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:23:21.193093    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:23:21.194669 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:23:21.194649    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:25:29.683128 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.683088    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"]
Jul 28 13:25:29.683705 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.683332    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container" containerID="cri-o://7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379" gracePeriod=30
Jul 28 13:25:29.683705 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.683390    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kube-rbac-proxy" containerID="cri-o://11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda" gracePeriod=30
Jul 28 13:25:29.754882 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.754841    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"]
Jul 28 13:25:29.755251 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.755236    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container"
Jul 28 13:25:29.755307 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.755255    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container"
Jul 28 13:25:29.755307 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.755295    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kube-rbac-proxy"
Jul 28 13:25:29.755307 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.755303    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kube-rbac-proxy"
Jul 28 13:25:29.755412 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.755363    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kube-rbac-proxy"
Jul 28 13:25:29.755412 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.755374    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="31815b80-5909-48c4-b48f-1060343f97be" containerName="kserve-container"
Jul 28 13:25:29.758443 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.758423    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:29.760587 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.760565    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-198f4-predictor-serving-cert\""
Jul 28 13:25:29.760854 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.760839    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-198f4-kube-rbac-proxy-sar-config\""
Jul 28 13:25:29.772115 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.772087    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"]
Jul 28 13:25:29.820078 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.820042    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252mz\" (UniqueName: \"kubernetes.io/projected/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-api-access-252mz\") pod \"success-200-isvc-198f4-predictor-5cddbfbc59-26t4q\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") " pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:29.820268 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.820088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a692e98-61ff-4a13-a13a-c0def4d56c9f-proxy-tls\") pod \"success-200-isvc-198f4-predictor-5cddbfbc59-26t4q\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") " pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:29.820268 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.820130    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-198f4-predictor-5cddbfbc59-26t4q\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") " pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:29.921033 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.920999    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252mz\" (UniqueName: \"kubernetes.io/projected/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-api-access-252mz\") pod \"success-200-isvc-198f4-predictor-5cddbfbc59-26t4q\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") " pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:29.921033 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.921044    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a692e98-61ff-4a13-a13a-c0def4d56c9f-proxy-tls\") pod \"success-200-isvc-198f4-predictor-5cddbfbc59-26t4q\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") " pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:29.921284 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.921081    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-198f4-predictor-5cddbfbc59-26t4q\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") " pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:29.921747 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.921726    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-198f4-predictor-5cddbfbc59-26t4q\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") " pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:29.923663 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.923641    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a692e98-61ff-4a13-a13a-c0def4d56c9f-proxy-tls\") pod \"success-200-isvc-198f4-predictor-5cddbfbc59-26t4q\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") " pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:29.931925 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:29.931904    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-252mz\" (UniqueName: \"kubernetes.io/projected/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-api-access-252mz\") pod \"success-200-isvc-198f4-predictor-5cddbfbc59-26t4q\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") " pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:30.069694 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:30.069654    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:30.208742 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:30.208704    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"]
Jul 28 13:25:30.212214 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:25:30.212179    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a692e98_61ff_4a13_a13a_c0def4d56c9f.slice/crio-d3887429e734acf312b81e4b72a9df6d15d8faad79a5dfa0db915cabcf9681e7 WatchSource:0}: Error finding container d3887429e734acf312b81e4b72a9df6d15d8faad79a5dfa0db915cabcf9681e7: Status 404 returned error can't find the container with id d3887429e734acf312b81e4b72a9df6d15d8faad79a5dfa0db915cabcf9681e7
Jul 28 13:25:30.606750 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:30.606664    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerID="11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda" exitCode=2
Jul 28 13:25:30.606750 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:30.606738    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" event={"ID":"7b0da8a2-05d2-4e64-ae26-3c6478791e82","Type":"ContainerDied","Data":"11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda"}
Jul 28 13:25:30.608621 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:30.608585    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" event={"ID":"3a692e98-61ff-4a13-a13a-c0def4d56c9f","Type":"ContainerStarted","Data":"7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507"}
Jul 28 13:25:30.608762 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:30.608653    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" event={"ID":"3a692e98-61ff-4a13-a13a-c0def4d56c9f","Type":"ContainerStarted","Data":"35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d"}
Jul 28 13:25:30.608762 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:30.608683    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" event={"ID":"3a692e98-61ff-4a13-a13a-c0def4d56c9f","Type":"ContainerStarted","Data":"d3887429e734acf312b81e4b72a9df6d15d8faad79a5dfa0db915cabcf9681e7"}
Jul 28 13:25:30.608762 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:30.608720    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:30.626662 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:30.626603    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podStartSLOduration=1.6265828519999999 podStartE2EDuration="1.626582852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:25:30.625827022 +0000 UTC m=+1929.948819963" watchObservedRunningTime="2026-07-28 13:25:30.626582852 +0000 UTC m=+1929.949575793"
Jul 28 13:25:31.612831 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:31.612800    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:31.614535 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:31.614505    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 13:25:32.174874 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:32.174828    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 28 13:25:32.180235 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:32.180201    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 13:25:32.617223 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:32.617179    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 13:25:33.042674 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.042649    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:25:33.147160 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.147125    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0da8a2-05d2-4e64-ae26-3c6478791e82-proxy-tls\") pod \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") "
Jul 28 13:25:33.147336 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.147208    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-rbac-proxy-sar-config\") pod \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") "
Jul 28 13:25:33.147336 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.147253    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qggx6\" (UniqueName: \"kubernetes.io/projected/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-api-access-qggx6\") pod \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\" (UID: \"7b0da8a2-05d2-4e64-ae26-3c6478791e82\") "
Jul 28 13:25:33.147565 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.147540    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b0da8a2-05d2-4e64-ae26-3c6478791e82" (UID: "7b0da8a2-05d2-4e64-ae26-3c6478791e82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:25:33.149455 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.149421    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-api-access-qggx6" (OuterVolumeSpecName: "kube-api-access-qggx6") pod "7b0da8a2-05d2-4e64-ae26-3c6478791e82" (UID: "7b0da8a2-05d2-4e64-ae26-3c6478791e82"). InnerVolumeSpecName "kube-api-access-qggx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:25:33.149578 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.149462    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0da8a2-05d2-4e64-ae26-3c6478791e82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b0da8a2-05d2-4e64-ae26-3c6478791e82" (UID: "7b0da8a2-05d2-4e64-ae26-3c6478791e82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:25:33.248152 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.248112    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:25:33.248152 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.248146    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qggx6\" (UniqueName: \"kubernetes.io/projected/7b0da8a2-05d2-4e64-ae26-3c6478791e82-kube-api-access-qggx6\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:25:33.248152 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.248157    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0da8a2-05d2-4e64-ae26-3c6478791e82-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:25:33.623049 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.623012    2572 generic.go:356] "Generic (PLEG): container finished" podID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerID="7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379" exitCode=0
Jul 28 13:25:33.623464 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.623065    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"
Jul 28 13:25:33.623464 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.623095    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" event={"ID":"7b0da8a2-05d2-4e64-ae26-3c6478791e82","Type":"ContainerDied","Data":"7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379"}
Jul 28 13:25:33.623464 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.623154    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r" event={"ID":"7b0da8a2-05d2-4e64-ae26-3c6478791e82","Type":"ContainerDied","Data":"54b74ef34928b6c463d3ca1bd478e714f69bb19b540c56eef060a168cfe51761"}
Jul 28 13:25:33.623464 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.623181    2572 scope.go:117] "RemoveContainer" containerID="11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda"
Jul 28 13:25:33.631884 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.631866    2572 scope.go:117] "RemoveContainer" containerID="7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379"
Jul 28 13:25:33.639697 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.639663    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"]
Jul 28 13:25:33.639770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.639731    2572 scope.go:117] "RemoveContainer" containerID="11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda"
Jul 28 13:25:33.640041 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:25:33.640019    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda\": container with ID starting with 11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda not found: ID does not exist" containerID="11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda"
Jul 28 13:25:33.640128 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.640051    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda"} err="failed to get container status \"11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda\": rpc error: code = NotFound desc = could not find container \"11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda\": container with ID starting with 11c3c1e73f7da49afc04a401e478667d645beb9fad759afc6f8f6827c4885cda not found: ID does not exist"
Jul 28 13:25:33.640128 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.640076    2572 scope.go:117] "RemoveContainer" containerID="7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379"
Jul 28 13:25:33.640361 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:25:33.640343    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379\": container with ID starting with 7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379 not found: ID does not exist" containerID="7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379"
Jul 28 13:25:33.640402 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.640366    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379"} err="failed to get container status \"7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379\": rpc error: code = NotFound desc = could not find container \"7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379\": container with ID starting with 7e5f6752c384ff946152dbe1bc626c8ec714d7d01b67bedfcee29aa89d631379 not found: ID does not exist"
Jul 28 13:25:33.643625 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:33.643605    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cae9-predictor-7fc49f875c-n572r"]
Jul 28 13:25:35.180553 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:35.180512    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" path="/var/lib/kubelet/pods/7b0da8a2-05d2-4e64-ae26-3c6478791e82/volumes"
Jul 28 13:25:37.621617 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:37.621586    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:25:37.622187 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:37.622162    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 13:25:47.623182 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:47.623136    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 13:25:57.622149 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:25:57.622106    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 13:26:05.479785 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.479745    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"]
Jul 28 13:26:05.480404 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.480058    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container" containerID="cri-o://55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8" gracePeriod=30
Jul 28 13:26:05.480404 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.480107    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kube-rbac-proxy" containerID="cri-o://9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b" gracePeriod=30
Jul 28 13:26:05.571437 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.571401    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"]
Jul 28 13:26:05.571770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.571752    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container"
Jul 28 13:26:05.571770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.571768    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container"
Jul 28 13:26:05.571898 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.571775    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kube-rbac-proxy"
Jul 28 13:26:05.571898 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.571781    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kube-rbac-proxy"
Jul 28 13:26:05.571898 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.571850    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kserve-container"
Jul 28 13:26:05.571898 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.571864    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0da8a2-05d2-4e64-ae26-3c6478791e82" containerName="kube-rbac-proxy"
Jul 28 13:26:05.575184 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.575160    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:05.577637 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.577613    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-18d3f-predictor-serving-cert\""
Jul 28 13:26:05.577637 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.577631    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-18d3f-kube-rbac-proxy-sar-config\""
Jul 28 13:26:05.583488 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.583457    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"]
Jul 28 13:26:05.613682 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.613649    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-proxy-tls\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:05.613864 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.613702    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:05.613864 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.613788    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmcq\" (UniqueName: \"kubernetes.io/projected/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-api-access-tbmcq\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:05.714851 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.714805    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-proxy-tls\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:05.715102 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.714874    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:05.715102 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.714923    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmcq\" (UniqueName: \"kubernetes.io/projected/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-api-access-tbmcq\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:05.715102 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:26:05.714982    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-serving-cert: secret "success-200-isvc-18d3f-predictor-serving-cert" not found
Jul 28 13:26:05.715102 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:26:05.715060    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-proxy-tls podName:f898e3b3-c8d9-4485-9e3d-0f9e72b6773b nodeName:}" failed. No retries permitted until 2026-07-28 13:26:06.215037 +0000 UTC m=+1965.538029937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-proxy-tls") pod "success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" (UID: "f898e3b3-c8d9-4485-9e3d-0f9e72b6773b") : secret "success-200-isvc-18d3f-predictor-serving-cert" not found
Jul 28 13:26:05.716266 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.716239    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:05.726721 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.726690    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmcq\" (UniqueName: \"kubernetes.io/projected/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-api-access-tbmcq\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:05.773451 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.773410    2572 generic.go:356] "Generic (PLEG): container finished" podID="6092f68a-0296-4036-ae75-020b8de66fc4" containerID="9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b" exitCode=2
Jul 28 13:26:05.773609 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:05.773485    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" event={"ID":"6092f68a-0296-4036-ae75-020b8de66fc4","Type":"ContainerDied","Data":"9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b"}
Jul 28 13:26:06.218188 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:06.218074    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-proxy-tls\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:06.220488 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:06.220467    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-proxy-tls\") pod \"success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") " pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:06.486821 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:06.486725    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:06.619959 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:06.619921    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"]
Jul 28 13:26:06.622638 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:26:06.622607    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf898e3b3_c8d9_4485_9e3d_0f9e72b6773b.slice/crio-4a37dcc01b77603976bbf7bc71cdeb6a506a0ffc1db40a29e9bb889ba2090eb0 WatchSource:0}: Error finding container 4a37dcc01b77603976bbf7bc71cdeb6a506a0ffc1db40a29e9bb889ba2090eb0: Status 404 returned error can't find the container with id 4a37dcc01b77603976bbf7bc71cdeb6a506a0ffc1db40a29e9bb889ba2090eb0
Jul 28 13:26:06.779962 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:06.779918    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" event={"ID":"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b","Type":"ContainerStarted","Data":"613a74b5cb66e6050d643884acdbe9ce637c1f314559574b5d2fb71264e6b8ac"}
Jul 28 13:26:06.780142 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:06.780004    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" event={"ID":"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b","Type":"ContainerStarted","Data":"1dd599ac6c19fc6de71cffaf1cb6d984455106422dd9fabb4e91a3ba5c0c4528"}
Jul 28 13:26:06.780142 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:06.780029    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" event={"ID":"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b","Type":"ContainerStarted","Data":"4a37dcc01b77603976bbf7bc71cdeb6a506a0ffc1db40a29e9bb889ba2090eb0"}
Jul 28 13:26:06.780142 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:06.780124    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:06.798439 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:06.798385    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" podStartSLOduration=1.7983692470000001 podStartE2EDuration="1.798369247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:26:06.797929811 +0000 UTC m=+1966.120922752" watchObservedRunningTime="2026-07-28 13:26:06.798369247 +0000 UTC m=+1966.121362237"
Jul 28 13:26:07.340475 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:07.340437    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 13:26:07.345834 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:07.345800    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 13:26:07.622865 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:07.622774    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 13:26:07.784681 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:07.784650    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:07.786009 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:07.785977    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 13:26:08.788674 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:08.788637    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 13:26:09.135304 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.135276    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:26:09.137167 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.137146    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6092f68a-0296-4036-ae75-020b8de66fc4-proxy-tls\") pod \"6092f68a-0296-4036-ae75-020b8de66fc4\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") "
Jul 28 13:26:09.137261 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.137202    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q7ng\" (UniqueName: \"kubernetes.io/projected/6092f68a-0296-4036-ae75-020b8de66fc4-kube-api-access-9q7ng\") pod \"6092f68a-0296-4036-ae75-020b8de66fc4\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") "
Jul 28 13:26:09.137261 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.137245    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6092f68a-0296-4036-ae75-020b8de66fc4-kube-rbac-proxy-sar-config\") pod \"6092f68a-0296-4036-ae75-020b8de66fc4\" (UID: \"6092f68a-0296-4036-ae75-020b8de66fc4\") "
Jul 28 13:26:09.137551 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.137531    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6092f68a-0296-4036-ae75-020b8de66fc4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6092f68a-0296-4036-ae75-020b8de66fc4" (UID: "6092f68a-0296-4036-ae75-020b8de66fc4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:26:09.139338 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.139310    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6092f68a-0296-4036-ae75-020b8de66fc4-kube-api-access-9q7ng" (OuterVolumeSpecName: "kube-api-access-9q7ng") pod "6092f68a-0296-4036-ae75-020b8de66fc4" (UID: "6092f68a-0296-4036-ae75-020b8de66fc4"). InnerVolumeSpecName "kube-api-access-9q7ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:26:09.139429 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.139386    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6092f68a-0296-4036-ae75-020b8de66fc4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6092f68a-0296-4036-ae75-020b8de66fc4" (UID: "6092f68a-0296-4036-ae75-020b8de66fc4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:26:09.237916 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.237878    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6092f68a-0296-4036-ae75-020b8de66fc4-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:26:09.237916 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.237910    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q7ng\" (UniqueName: \"kubernetes.io/projected/6092f68a-0296-4036-ae75-020b8de66fc4-kube-api-access-9q7ng\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:26:09.237916 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.237920    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6092f68a-0296-4036-ae75-020b8de66fc4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:26:09.794287 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.794251    2572 generic.go:356] "Generic (PLEG): container finished" podID="6092f68a-0296-4036-ae75-020b8de66fc4" containerID="55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8" exitCode=0
Jul 28 13:26:09.794779 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.794325    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"
Jul 28 13:26:09.794779 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.794336    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" event={"ID":"6092f68a-0296-4036-ae75-020b8de66fc4","Type":"ContainerDied","Data":"55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8"}
Jul 28 13:26:09.794779 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.794420    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p" event={"ID":"6092f68a-0296-4036-ae75-020b8de66fc4","Type":"ContainerDied","Data":"0f26cd3bd0c73168d16e3c5bd7a0bdcf02884d30746beda3c341468e62f342b6"}
Jul 28 13:26:09.794779 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.794463    2572 scope.go:117] "RemoveContainer" containerID="9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b"
Jul 28 13:26:09.803169 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.803153    2572 scope.go:117] "RemoveContainer" containerID="55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8"
Jul 28 13:26:09.810927 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.810907    2572 scope.go:117] "RemoveContainer" containerID="9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b"
Jul 28 13:26:09.811266 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:26:09.811243    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b\": container with ID starting with 9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b not found: ID does not exist" containerID="9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b"
Jul 28 13:26:09.811365 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.811270    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b"} err="failed to get container status \"9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b\": rpc error: code = NotFound desc = could not find container \"9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b\": container with ID starting with 9e8ab47d7e82e2b1a7e4f7b9ec1eaaf64d21c23fc1f8cae485a39639c2e8d70b not found: ID does not exist"
Jul 28 13:26:09.811365 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.811287    2572 scope.go:117] "RemoveContainer" containerID="55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8"
Jul 28 13:26:09.811562 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:26:09.811532    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8\": container with ID starting with 55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8 not found: ID does not exist" containerID="55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8"
Jul 28 13:26:09.811684 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.811571    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8"} err="failed to get container status \"55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8\": rpc error: code = NotFound desc = could not find container \"55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8\": container with ID starting with 55b3b5916e90b0a6ecce7cca32108f5d0858542c4df21d5c1e20fd17b93a0dd8 not found: ID does not exist"
Jul 28 13:26:09.812723 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.812695    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"]
Jul 28 13:26:09.815605 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:09.815584    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dd548-predictor-d86d86669-w2r2p"]
Jul 28 13:26:11.180230 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:11.180196    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" path="/var/lib/kubelet/pods/6092f68a-0296-4036-ae75-020b8de66fc4/volumes"
Jul 28 13:26:13.793775 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:13.793698    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:13.794197 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:13.794170    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 13:26:17.623125 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:17.623095    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:26:23.795192 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:23.795148    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 13:26:33.794859 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:33.794809    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 13:26:39.934454 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:39.934418    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"]
Jul 28 13:26:39.935000 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:39.934758    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container" containerID="cri-o://35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d" gracePeriod=30
Jul 28 13:26:39.935000 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:39.934885    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kube-rbac-proxy" containerID="cri-o://7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507" gracePeriod=30
Jul 28 13:26:40.020209 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.020170    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"]
Jul 28 13:26:40.020604 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.020586    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kube-rbac-proxy"
Jul 28 13:26:40.020695 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.020607    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kube-rbac-proxy"
Jul 28 13:26:40.020695 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.020652    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container"
Jul 28 13:26:40.020695 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.020662    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container"
Jul 28 13:26:40.020850 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.020742    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kserve-container"
Jul 28 13:26:40.020850 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.020760    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6092f68a-0296-4036-ae75-020b8de66fc4" containerName="kube-rbac-proxy"
Jul 28 13:26:40.024283 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.024261    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.026279 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.026255    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b971b-predictor-serving-cert\""
Jul 28 13:26:40.026385 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.026257    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b971b-kube-rbac-proxy-sar-config\""
Jul 28 13:26:40.033560 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.033533    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"]
Jul 28 13:26:40.076771 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.076718    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c436d7e-34f2-4fde-b715-38f6e0665a6c-proxy-tls\") pod \"success-200-isvc-b971b-predictor-694db9976f-m4mbc\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") " pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.076927 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.076800    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnh8s\" (UniqueName: \"kubernetes.io/projected/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-api-access-lnh8s\") pod \"success-200-isvc-b971b-predictor-694db9976f-m4mbc\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") " pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.076927 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.076870    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b971b-predictor-694db9976f-m4mbc\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") " pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.177464 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.177429    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b971b-predictor-694db9976f-m4mbc\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") " pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.177654 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.177507    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c436d7e-34f2-4fde-b715-38f6e0665a6c-proxy-tls\") pod \"success-200-isvc-b971b-predictor-694db9976f-m4mbc\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") " pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.177654 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.177537    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnh8s\" (UniqueName: \"kubernetes.io/projected/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-api-access-lnh8s\") pod \"success-200-isvc-b971b-predictor-694db9976f-m4mbc\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") " pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.178134 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.178109    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b971b-predictor-694db9976f-m4mbc\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") " pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.180024 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.180003    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c436d7e-34f2-4fde-b715-38f6e0665a6c-proxy-tls\") pod \"success-200-isvc-b971b-predictor-694db9976f-m4mbc\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") " pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.185178 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.185128    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnh8s\" (UniqueName: \"kubernetes.io/projected/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-api-access-lnh8s\") pod \"success-200-isvc-b971b-predictor-694db9976f-m4mbc\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") " pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.335937 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.335898    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.481983 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.481876    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"]
Jul 28 13:26:40.485271 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:26:40.485247    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c436d7e_34f2_4fde_b715_38f6e0665a6c.slice/crio-4779ee3d37b46359d9eb67effed21d1c1d0343b6e4b4c100a2458c4c31418897 WatchSource:0}: Error finding container 4779ee3d37b46359d9eb67effed21d1c1d0343b6e4b4c100a2458c4c31418897: Status 404 returned error can't find the container with id 4779ee3d37b46359d9eb67effed21d1c1d0343b6e4b4c100a2458c4c31418897
Jul 28 13:26:40.936638 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.936598    2572 generic.go:356] "Generic (PLEG): container finished" podID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerID="7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507" exitCode=2
Jul 28 13:26:40.936638 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.936634    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" event={"ID":"3a692e98-61ff-4a13-a13a-c0def4d56c9f","Type":"ContainerDied","Data":"7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507"}
Jul 28 13:26:40.938402 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.938376    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" event={"ID":"5c436d7e-34f2-4fde-b715-38f6e0665a6c","Type":"ContainerStarted","Data":"c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3"}
Jul 28 13:26:40.938538 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.938421    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" event={"ID":"5c436d7e-34f2-4fde-b715-38f6e0665a6c","Type":"ContainerStarted","Data":"a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08"}
Jul 28 13:26:40.938538 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.938445    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" event={"ID":"5c436d7e-34f2-4fde-b715-38f6e0665a6c","Type":"ContainerStarted","Data":"4779ee3d37b46359d9eb67effed21d1c1d0343b6e4b4c100a2458c4c31418897"}
Jul 28 13:26:40.938538 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.938501    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:40.956769 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:40.956725    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podStartSLOduration=0.956711727 podStartE2EDuration="956.711727ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:26:40.955176565 +0000 UTC m=+2000.278169523" watchObservedRunningTime="2026-07-28 13:26:40.956711727 +0000 UTC m=+2000.279704667"
Jul 28 13:26:41.943400 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:41.943362    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:41.944790 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:41.944760    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:26:42.617711 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:42.617660    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 28 13:26:42.947771 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:42.947678    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:26:43.315529 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.315503    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:26:43.410028 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.409994    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a692e98-61ff-4a13-a13a-c0def4d56c9f-proxy-tls\") pod \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") "
Jul 28 13:26:43.410206 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.410046    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-rbac-proxy-sar-config\") pod \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") "
Jul 28 13:26:43.410206 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.410115    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-252mz\" (UniqueName: \"kubernetes.io/projected/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-api-access-252mz\") pod \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\" (UID: \"3a692e98-61ff-4a13-a13a-c0def4d56c9f\") "
Jul 28 13:26:43.410498 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.410457    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a692e98-61ff-4a13-a13a-c0def4d56c9f" (UID: "3a692e98-61ff-4a13-a13a-c0def4d56c9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:26:43.412234 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.412201    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-api-access-252mz" (OuterVolumeSpecName: "kube-api-access-252mz") pod "3a692e98-61ff-4a13-a13a-c0def4d56c9f" (UID: "3a692e98-61ff-4a13-a13a-c0def4d56c9f"). InnerVolumeSpecName "kube-api-access-252mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:26:43.412332 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.412306    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a692e98-61ff-4a13-a13a-c0def4d56c9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a692e98-61ff-4a13-a13a-c0def4d56c9f" (UID: "3a692e98-61ff-4a13-a13a-c0def4d56c9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:26:43.511343 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.511309    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252mz\" (UniqueName: \"kubernetes.io/projected/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-api-access-252mz\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:26:43.511343 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.511345    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a692e98-61ff-4a13-a13a-c0def4d56c9f-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:26:43.511541 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.511358    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a692e98-61ff-4a13-a13a-c0def4d56c9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:26:43.794754 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.794668    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 13:26:43.953112 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.953069    2572 generic.go:356] "Generic (PLEG): container finished" podID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerID="35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d" exitCode=0
Jul 28 13:26:43.953527 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.953122    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"
Jul 28 13:26:43.953527 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.953115    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" event={"ID":"3a692e98-61ff-4a13-a13a-c0def4d56c9f","Type":"ContainerDied","Data":"35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d"}
Jul 28 13:26:43.953527 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.953252    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q" event={"ID":"3a692e98-61ff-4a13-a13a-c0def4d56c9f","Type":"ContainerDied","Data":"d3887429e734acf312b81e4b72a9df6d15d8faad79a5dfa0db915cabcf9681e7"}
Jul 28 13:26:43.953527 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.953280    2572 scope.go:117] "RemoveContainer" containerID="7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507"
Jul 28 13:26:43.962266 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.962245    2572 scope.go:117] "RemoveContainer" containerID="35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d"
Jul 28 13:26:43.970024 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.969932    2572 scope.go:117] "RemoveContainer" containerID="7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507"
Jul 28 13:26:43.970308 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:26:43.970289    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507\": container with ID starting with 7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507 not found: ID does not exist" containerID="7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507"
Jul 28 13:26:43.970347 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.970318    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507"} err="failed to get container status \"7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507\": rpc error: code = NotFound desc = could not find container \"7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507\": container with ID starting with 7d323536f193e191e5e302f4c6797967cd853bf89c7372f833a325a70fb98507 not found: ID does not exist"
Jul 28 13:26:43.970347 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.970335    2572 scope.go:117] "RemoveContainer" containerID="35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d"
Jul 28 13:26:43.970539 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:26:43.970524    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d\": container with ID starting with 35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d not found: ID does not exist" containerID="35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d"
Jul 28 13:26:43.970582 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.970544    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d"} err="failed to get container status \"35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d\": rpc error: code = NotFound desc = could not find container \"35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d\": container with ID starting with 35bf06e7b79ac2a0d784fe970d2f4b57cdad94ccb4243f4f30efad34fa6f024d not found: ID does not exist"
Jul 28 13:26:43.976687 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.976664    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"]
Jul 28 13:26:43.982622 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:43.982600    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-198f4-predictor-5cddbfbc59-26t4q"]
Jul 28 13:26:45.183170 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:45.181607    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" path="/var/lib/kubelet/pods/3a692e98-61ff-4a13-a13a-c0def4d56c9f/volumes"
Jul 28 13:26:47.952573 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:47.952543    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:26:47.953042 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:47.953016    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:26:53.795106 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:53.795072    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:26:57.954030 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:26:57.953983    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:27:07.953092 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:27:07.953047    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:27:17.953605 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:27:17.953565    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:27:27.953924 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:27:27.953893    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:28:21.224207 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:28:21.224174    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:28:21.226905 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:28:21.226887    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:33:21.253301 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:33:21.253185    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:33:21.257186 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:33:21.256887    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:35:54.898424 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:54.898387    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"]
Jul 28 13:35:54.898987 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:54.898649    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container" containerID="cri-o://a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08" gracePeriod=30
Jul 28 13:35:54.898987 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:54.898685    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kube-rbac-proxy" containerID="cri-o://c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3" gracePeriod=30
Jul 28 13:35:55.377728 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:55.377693    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerID="c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3" exitCode=2
Jul 28 13:35:55.377918 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:55.377766    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" event={"ID":"5c436d7e-34f2-4fde-b715-38f6e0665a6c","Type":"ContainerDied","Data":"c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3"}
Jul 28 13:35:57.948920 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:57.948881    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 28 13:35:57.953238 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:57.953205    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:35:58.159292 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.159262    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:35:58.259447 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.259417    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c436d7e-34f2-4fde-b715-38f6e0665a6c-proxy-tls\") pod \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") "
Jul 28 13:35:58.259606 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.259464    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-rbac-proxy-sar-config\") pod \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") "
Jul 28 13:35:58.259606 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.259484    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnh8s\" (UniqueName: \"kubernetes.io/projected/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-api-access-lnh8s\") pod \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\" (UID: \"5c436d7e-34f2-4fde-b715-38f6e0665a6c\") "
Jul 28 13:35:58.260005 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.259855    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c436d7e-34f2-4fde-b715-38f6e0665a6c" (UID: "5c436d7e-34f2-4fde-b715-38f6e0665a6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:35:58.261633 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.261604    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c436d7e-34f2-4fde-b715-38f6e0665a6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c436d7e-34f2-4fde-b715-38f6e0665a6c" (UID: "5c436d7e-34f2-4fde-b715-38f6e0665a6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:35:58.261731 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.261636    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-api-access-lnh8s" (OuterVolumeSpecName: "kube-api-access-lnh8s") pod "5c436d7e-34f2-4fde-b715-38f6e0665a6c" (UID: "5c436d7e-34f2-4fde-b715-38f6e0665a6c"). InnerVolumeSpecName "kube-api-access-lnh8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:35:58.360300 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.360259    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:35:58.360300 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.360291    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnh8s\" (UniqueName: \"kubernetes.io/projected/5c436d7e-34f2-4fde-b715-38f6e0665a6c-kube-api-access-lnh8s\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:35:58.360300 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.360302    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c436d7e-34f2-4fde-b715-38f6e0665a6c-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:35:58.393275 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.393235    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerID="a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08" exitCode=0
Jul 28 13:35:58.393430 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.393280    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"
Jul 28 13:35:58.393430 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.393294    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" event={"ID":"5c436d7e-34f2-4fde-b715-38f6e0665a6c","Type":"ContainerDied","Data":"a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08"}
Jul 28 13:35:58.393430 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.393360    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc" event={"ID":"5c436d7e-34f2-4fde-b715-38f6e0665a6c","Type":"ContainerDied","Data":"4779ee3d37b46359d9eb67effed21d1c1d0343b6e4b4c100a2458c4c31418897"}
Jul 28 13:35:58.393430 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.393393    2572 scope.go:117] "RemoveContainer" containerID="c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3"
Jul 28 13:35:58.402340 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.402318    2572 scope.go:117] "RemoveContainer" containerID="a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08"
Jul 28 13:35:58.409704 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.409679    2572 scope.go:117] "RemoveContainer" containerID="c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3"
Jul 28 13:35:58.410005 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:35:58.409987    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3\": container with ID starting with c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3 not found: ID does not exist" containerID="c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3"
Jul 28 13:35:58.410071 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.410013    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3"} err="failed to get container status \"c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3\": rpc error: code = NotFound desc = could not find container \"c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3\": container with ID starting with c762432d43c6b6a38730883b218a345e53224a330d244a57a669e3d02594e3f3 not found: ID does not exist"
Jul 28 13:35:58.410071 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.410031    2572 scope.go:117] "RemoveContainer" containerID="a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08"
Jul 28 13:35:58.410261 ip-10-0-142-83 kubenswrapper[2572]: E0728 13:35:58.410239    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08\": container with ID starting with a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08 not found: ID does not exist" containerID="a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08"
Jul 28 13:35:58.410320 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.410273    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08"} err="failed to get container status \"a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08\": rpc error: code = NotFound desc = could not find container \"a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08\": container with ID starting with a1f00c0338f99369e2a76dab6d04c58420b9f5ab1026762f2a77890f9d74fe08 not found: ID does not exist"
Jul 28 13:35:58.413925 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.413901    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"]
Jul 28 13:35:58.418572 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:58.418546    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b971b-predictor-694db9976f-m4mbc"]
Jul 28 13:35:59.180659 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:35:59.180621    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" path="/var/lib/kubelet/pods/5c436d7e-34f2-4fde-b715-38f6e0665a6c/volumes"
Jul 28 13:38:21.280235 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:38:21.280199    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:38:21.283709 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:38:21.283685    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:43:21.304437 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:21.304328    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:43:21.312133 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:21.312112    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:43:25.128900 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:25.128849    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"]
Jul 28 13:43:25.129422 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:25.129204    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container" containerID="cri-o://1dd599ac6c19fc6de71cffaf1cb6d984455106422dd9fabb4e91a3ba5c0c4528" gracePeriod=30
Jul 28 13:43:25.129422 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:25.129269    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kube-rbac-proxy" containerID="cri-o://613a74b5cb66e6050d643884acdbe9ce637c1f314559574b5d2fb71264e6b8ac" gracePeriod=30
Jul 28 13:43:25.314413 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:25.314376    2572 generic.go:356] "Generic (PLEG): container finished" podID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerID="613a74b5cb66e6050d643884acdbe9ce637c1f314559574b5d2fb71264e6b8ac" exitCode=2
Jul 28 13:43:25.314585 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:25.314441    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" event={"ID":"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b","Type":"ContainerDied","Data":"613a74b5cb66e6050d643884acdbe9ce637c1f314559574b5d2fb71264e6b8ac"}
Jul 28 13:43:28.329409 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.329378    2572 generic.go:356] "Generic (PLEG): container finished" podID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerID="1dd599ac6c19fc6de71cffaf1cb6d984455106422dd9fabb4e91a3ba5c0c4528" exitCode=0
Jul 28 13:43:28.329803 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.329455    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" event={"ID":"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b","Type":"ContainerDied","Data":"1dd599ac6c19fc6de71cffaf1cb6d984455106422dd9fabb4e91a3ba5c0c4528"}
Jul 28 13:43:28.378823 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.378800    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:43:28.539082 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.539046    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-rbac-proxy-sar-config\") pod \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") "
Jul 28 13:43:28.539305 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.539109    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbmcq\" (UniqueName: \"kubernetes.io/projected/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-api-access-tbmcq\") pod \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") "
Jul 28 13:43:28.539305 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.539139    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-proxy-tls\") pod \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\" (UID: \"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b\") "
Jul 28 13:43:28.539460 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.539437    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" (UID: "f898e3b3-c8d9-4485-9e3d-0f9e72b6773b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:43:28.541384 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.541354    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" (UID: "f898e3b3-c8d9-4485-9e3d-0f9e72b6773b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:43:28.541488 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.541419    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-api-access-tbmcq" (OuterVolumeSpecName: "kube-api-access-tbmcq") pod "f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" (UID: "f898e3b3-c8d9-4485-9e3d-0f9e72b6773b"). InnerVolumeSpecName "kube-api-access-tbmcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:43:28.640286 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.640242    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:43:28.640286 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.640276    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmcq\" (UniqueName: \"kubernetes.io/projected/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-kube-api-access-tbmcq\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:43:28.640286 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:28.640286    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b-proxy-tls\") on node \"ip-10-0-142-83.ec2.internal\" DevicePath \"\""
Jul 28 13:43:29.336259 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:29.336225    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb" event={"ID":"f898e3b3-c8d9-4485-9e3d-0f9e72b6773b","Type":"ContainerDied","Data":"4a37dcc01b77603976bbf7bc71cdeb6a506a0ffc1db40a29e9bb889ba2090eb0"}
Jul 28 13:43:29.336259 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:29.336261    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"
Jul 28 13:43:29.336774 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:29.336294    2572 scope.go:117] "RemoveContainer" containerID="613a74b5cb66e6050d643884acdbe9ce637c1f314559574b5d2fb71264e6b8ac"
Jul 28 13:43:29.344671 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:29.344653    2572 scope.go:117] "RemoveContainer" containerID="1dd599ac6c19fc6de71cffaf1cb6d984455106422dd9fabb4e91a3ba5c0c4528"
Jul 28 13:43:29.353864 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:29.353840    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"]
Jul 28 13:43:29.359271 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:29.359215    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-18d3f-predictor-87fcb7fff-q5vjb"]
Jul 28 13:43:31.180206 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:31.180162    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" path="/var/lib/kubelet/pods/f898e3b3-c8d9-4485-9e3d-0f9e72b6773b/volumes"
Jul 28 13:43:59.025091 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:59.025058    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p6smg_64386eb9-8899-410e-adf0-3f21a2a226b0/global-pull-secret-syncer/0.log"
Jul 28 13:43:59.120893 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:59.120862    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q7rl6_b52c3e58-14e8-442c-b277-6ba3718e5682/konnectivity-agent/0.log"
Jul 28 13:43:59.236002 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:43:59.235971    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-83.ec2.internal_5e36d7aafc98cad33ea701597790b088/haproxy/0.log"
Jul 28 13:44:03.680863 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:03.680827    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-8kwgx_4c4f466e-4360-449f-8ab8-da1732648943/cluster-monitoring-operator/0.log"
Jul 28 13:44:03.911143 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:03.911109    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9k8tg_79b89a03-eb00-4600-8dbb-14315a23d3a3/node-exporter/0.log"
Jul 28 13:44:03.933578 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:03.933462    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9k8tg_79b89a03-eb00-4600-8dbb-14315a23d3a3/kube-rbac-proxy/0.log"
Jul 28 13:44:03.959220 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:03.959197    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9k8tg_79b89a03-eb00-4600-8dbb-14315a23d3a3/init-textfile/0.log"
Jul 28 13:44:04.371441 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:04.371397    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-2jxtg_04bddd1a-5db7-46dc-a8bc-f725240331e9/prometheus-operator-admission-webhook/0.log"
Jul 28 13:44:05.935979 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:05.935935    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-9dkd5_25ff57f5-17a0-46b8-9fa6-6e6a05de1ded/networking-console-plugin/0.log"
Jul 28 13:44:06.416276 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:06.416246    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/2.log"
Jul 28 13:44:06.423661 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:06.423626    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-zmt5l_289ee42c-3969-4bcb-90f5-c0fe0ea9562f/console-operator/3.log"
Jul 28 13:44:07.274232 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274194    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"]
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274505    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274516    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274523    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kube-rbac-proxy"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274528    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kube-rbac-proxy"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274542    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kube-rbac-proxy"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274547    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kube-rbac-proxy"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274558    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274564    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274574    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274579    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274585    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kube-rbac-proxy"
Jul 28 13:44:07.274615 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274590    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kube-rbac-proxy"
Jul 28 13:44:07.274998 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274635    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kube-rbac-proxy"
Jul 28 13:44:07.274998 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274642    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c436d7e-34f2-4fde-b715-38f6e0665a6c" containerName="kserve-container"
Jul 28 13:44:07.274998 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274651    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kube-rbac-proxy"
Jul 28 13:44:07.274998 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274658    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f898e3b3-c8d9-4485-9e3d-0f9e72b6773b" containerName="kserve-container"
Jul 28 13:44:07.274998 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274666    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kube-rbac-proxy"
Jul 28 13:44:07.274998 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.274671    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a692e98-61ff-4a13-a13a-c0def4d56c9f" containerName="kserve-container"
Jul 28 13:44:07.277631 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.277611    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.279687 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.279660    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95qqx\"/\"openshift-service-ca.crt\""
Jul 28 13:44:07.279687 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.279677    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-95qqx\"/\"default-dockercfg-7wr6f\""
Jul 28 13:44:07.280182 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.280167    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95qqx\"/\"kube-root-ca.crt\""
Jul 28 13:44:07.285011 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.284988    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"]
Jul 28 13:44:07.341649 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.341612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-proc\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.341649 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.341654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-podres\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.341975 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.341678    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-lib-modules\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.341975 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.341750    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lhf\" (UniqueName: \"kubernetes.io/projected/067c1e98-0d69-4e7c-9e00-576d6d05c57b-kube-api-access-q5lhf\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.341975 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.341831    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-sys\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.359084 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.359051    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-q4x7f_f46daa6b-8762-4b6e-967f-2ae0d994f592/volume-data-source-validator/0.log"
Jul 28 13:44:07.442511 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.442472    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-sys\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.442770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.442540    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-proc\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.442770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.442567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-podres\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.442770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.442600    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-lib-modules\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.442770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.442610    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-sys\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.442770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.442627    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lhf\" (UniqueName: \"kubernetes.io/projected/067c1e98-0d69-4e7c-9e00-576d6d05c57b-kube-api-access-q5lhf\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.442770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.442679    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-proc\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.442770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.442744    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-lib-modules\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.442770 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.442749    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/067c1e98-0d69-4e7c-9e00-576d6d05c57b-podres\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.451065 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.451046    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lhf\" (UniqueName: \"kubernetes.io/projected/067c1e98-0d69-4e7c-9e00-576d6d05c57b-kube-api-access-q5lhf\") pod \"perf-node-gather-daemonset-hkmbf\" (UID: \"067c1e98-0d69-4e7c-9e00-576d6d05c57b\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.587843 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.587751    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:07.721972 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:07.721919    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"]
Jul 28 13:44:07.725281 ip-10-0-142-83 kubenswrapper[2572]: W0728 13:44:07.725205    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod067c1e98_0d69_4e7c_9e00_576d6d05c57b.slice/crio-d42c59333e0f1a3c8a7f1da5cfdf9ddfea15ca048c3f679af30deefe0724efec WatchSource:0}: Error finding container d42c59333e0f1a3c8a7f1da5cfdf9ddfea15ca048c3f679af30deefe0724efec: Status 404 returned error can't find the container with id d42c59333e0f1a3c8a7f1da5cfdf9ddfea15ca048c3f679af30deefe0724efec
Jul 28 13:44:08.290459 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:08.290434    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c2st_7f5a807c-9e19-4693-acc3-20e97affcb82/dns/0.log"
Jul 28 13:44:08.310043 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:08.310015    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c2st_7f5a807c-9e19-4693-acc3-20e97affcb82/kube-rbac-proxy/0.log"
Jul 28 13:44:08.406276 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:08.406238    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-szhdp_ad991f88-1a5a-4745-b3c6-7f00b9f4ee96/dns-node-resolver/0.log"
Jul 28 13:44:08.506527 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:08.506487    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf" event={"ID":"067c1e98-0d69-4e7c-9e00-576d6d05c57b","Type":"ContainerStarted","Data":"3963a48ec407b99bfc0ace3d831c2905abe0e5ced4593199dd06e9130471d9b5"}
Jul 28 13:44:08.506719 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:08.506553    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf" event={"ID":"067c1e98-0d69-4e7c-9e00-576d6d05c57b","Type":"ContainerStarted","Data":"d42c59333e0f1a3c8a7f1da5cfdf9ddfea15ca048c3f679af30deefe0724efec"}
Jul 28 13:44:08.506719 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:08.506692    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:08.522437 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:08.522374    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf" podStartSLOduration=1.5223553459999999 podStartE2EDuration="1.522355346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:08.520891621 +0000 UTC m=+3047.843884560" watchObservedRunningTime="2026-07-28 13:44:08.522355346 +0000 UTC m=+3047.845348287"
Jul 28 13:44:08.929992 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:08.929959    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ksgqt_c0e02d46-2f01-4d45-b048-f0dc22cfad61/node-ca/0.log"
Jul 28 13:44:09.663241 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:09.663191    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8b6d76f97-jh5t9_0d851530-6f45-4c04-96c3-c95bbdf2241e/router/0.log"
Jul 28 13:44:09.970378 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:09.970296    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5s4dp_c1ea0458-166f-4cab-a151-ae16ad59c313/serve-healthcheck-canary/0.log"
Jul 28 13:44:10.364579 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:10.364541    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-rgl5v_8f6ec1da-be3a-4780-9199-dc20768d96e2/insights-operator/0.log"
Jul 28 13:44:10.366352 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:10.366312    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-rgl5v_8f6ec1da-be3a-4780-9199-dc20768d96e2/insights-operator/1.log"
Jul 28 13:44:10.384957 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:10.384920    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lcbtm_08ef64bf-f32f-4215-8259-6a70cf9f3f84/kube-rbac-proxy/0.log"
Jul 28 13:44:10.403864 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:10.403811    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lcbtm_08ef64bf-f32f-4215-8259-6a70cf9f3f84/exporter/0.log"
Jul 28 13:44:10.429401 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:10.429370    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lcbtm_08ef64bf-f32f-4215-8259-6a70cf9f3f84/extractor/0.log"
Jul 28 13:44:12.782537 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:12.782504    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-ff96d5959-qx77q_50ff5da8-7c20-41a5-9260-6623b5af0118/manager/0.log"
Jul 28 13:44:12.840088 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:12.840059    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-qr44z_d72388a7-93f1-42dc-aa06-2aac5e985452/server/0.log"
Jul 28 13:44:12.990872 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:12.990842    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-vcf8f_b9369769-2bd9-4aba-b5b7-5d1e052ead91/manager/0.log"
Jul 28 13:44:14.521264 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:14.521191    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-hkmbf"
Jul 28 13:44:16.913245 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:16.913121    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-rb7cr_3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab/migrator/0.log"
Jul 28 13:44:16.940287 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:16.940259    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-rb7cr_3ae1eed1-71ad-4bb2-9b98-7c0323b2e7ab/graceful-termination/0.log"
Jul 28 13:44:17.424269 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:17.424237    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj_fd4afe72-14f5-4172-b6e9-3eee63e1c5f6/kube-storage-version-migrator-operator/1.log"
Jul 28 13:44:17.425890 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:17.425863    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-l9hfj_fd4afe72-14f5-4172-b6e9-3eee63e1c5f6/kube-storage-version-migrator-operator/0.log"
Jul 28 13:44:18.781678 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:18.781649    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mpcpt_bd90f16b-a738-49f8-99f8-ac9981959493/kube-multus-additional-cni-plugins/0.log"
Jul 28 13:44:18.803601 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:18.803574    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mpcpt_bd90f16b-a738-49f8-99f8-ac9981959493/egress-router-binary-copy/0.log"
Jul 28 13:44:18.823725 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:18.823695    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mpcpt_bd90f16b-a738-49f8-99f8-ac9981959493/cni-plugins/0.log"
Jul 28 13:44:18.842564 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:18.842539    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mpcpt_bd90f16b-a738-49f8-99f8-ac9981959493/bond-cni-plugin/0.log"
Jul 28 13:44:18.874674 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:18.874643    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mpcpt_bd90f16b-a738-49f8-99f8-ac9981959493/routeoverride-cni/0.log"
Jul 28 13:44:18.896080 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:18.896054    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mpcpt_bd90f16b-a738-49f8-99f8-ac9981959493/whereabouts-cni-bincopy/0.log"
Jul 28 13:44:18.915241 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:18.915212    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mpcpt_bd90f16b-a738-49f8-99f8-ac9981959493/whereabouts-cni/0.log"
Jul 28 13:44:18.950369 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:18.950335    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bwszv_f6b221b1-965c-43c6-aacb-9ba92b12d4a9/kube-multus/0.log"
Jul 28 13:44:19.037129 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:19.037045    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k4prs_43fa5143-75d6-4f79-88d0-97b5a5ecf09c/network-metrics-daemon/0.log"
Jul 28 13:44:19.055443 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:19.055413    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k4prs_43fa5143-75d6-4f79-88d0-97b5a5ecf09c/kube-rbac-proxy/0.log"
Jul 28 13:44:20.261427 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:20.261391    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4krx_66460849-0183-481b-a998-ac5bc9c81e35/ovn-controller/0.log"
Jul 28 13:44:20.278074 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:20.278035    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4krx_66460849-0183-481b-a998-ac5bc9c81e35/ovn-acl-logging/0.log"
Jul 28 13:44:20.302531 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:20.302499    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4krx_66460849-0183-481b-a998-ac5bc9c81e35/kube-rbac-proxy-node/0.log"
Jul 28 13:44:20.322587 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:20.322551    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4krx_66460849-0183-481b-a998-ac5bc9c81e35/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 13:44:20.338213 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:20.338180    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4krx_66460849-0183-481b-a998-ac5bc9c81e35/northd/0.log"
Jul 28 13:44:20.357234 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:20.357209    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4krx_66460849-0183-481b-a998-ac5bc9c81e35/nbdb/0.log"
Jul 28 13:44:20.380291 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:20.380262    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4krx_66460849-0183-481b-a998-ac5bc9c81e35/sbdb/0.log"
Jul 28 13:44:20.445039 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:20.445004    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4krx_66460849-0183-481b-a998-ac5bc9c81e35/ovnkube-controller/0.log"
Jul 28 13:44:21.646195 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:21.646161    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-gbqpx_b43f2f48-3db6-44ce-a879-e31a97f719d9/check-endpoints/0.log"
Jul 28 13:44:21.713213 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:21.713125    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-lkksh_64cdd13d-520d-4f12-86fa-a5f99d362fba/network-check-target-container/0.log"
Jul 28 13:44:22.623270 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:22.623238    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-tvzl9_11b94fe7-fd89-41af-b09b-79cfcb8afad8/iptables-alerter/0.log"
Jul 28 13:44:23.294235 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:23.294198    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-stdct_8581eb58-b73d-4cc3-983c-585bb739933c/tuned/0.log"
Jul 28 13:44:24.967858 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:24.967820    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6748489656-4wxjt_9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c/cluster-samples-operator/0.log"
Jul 28 13:44:24.983684 ip-10-0-142-83 kubenswrapper[2572]: I0728 13:44:24.983657    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6748489656-4wxjt_9d2d901c-fd6e-4c80-ae9a-4a3cda41c08c/cluster-samples-operator-watch/0.log"