Jul 09 03:59:53.509493 ip-10-0-142-5 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 03:59:53.910323 ip-10-0-142-5 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 03:59:53.910323 ip-10-0-142-5 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 03:59:53.910323 ip-10-0-142-5 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 03:59:53.910323 ip-10-0-142-5 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 03:59:53.910323 ip-10-0-142-5 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 03:59:53.910990 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.910381    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 03:59:53.918147 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918132    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 03:59:53.918147 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918147    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918151    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918156    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918159    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918162    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918165    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918167    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918170    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918173    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918181    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918184    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918187    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918190    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918192    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918195    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918197    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918200    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918202    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918216    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918221    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 03:59:53.918240 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918225    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918228    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918230    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918235    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918238    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918242    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918245    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918249    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918252    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918255    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918258    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918260    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918263    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918266    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918268    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918271    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918273    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918276    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918279    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918281    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 03:59:53.918763 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918284    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918286    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918289    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918291    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918294    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918296    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918298    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918301    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918303    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918306    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918308    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918310    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918313    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918315    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918318    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918321    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918324    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918326    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918329    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918331    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 03:59:53.919259 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918333    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918336    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918338    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918342    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918344    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918347    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918351    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918355    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918359    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918362    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918365    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918368    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918371    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918374    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918377    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918379    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918382    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918384    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918387    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 03:59:53.919739 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918389    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918392    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918394    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918397    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918399    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918401    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918404    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918407    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918410    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918413    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918415    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918418    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918421    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918424    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918426    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918429    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918431    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918433    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.918436    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920158    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 03:59:53.920195 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920167    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920170    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920173    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920176    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920181    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920185    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920188    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920191    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920193    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920196    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920198    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920201    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920215    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920219    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920221    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920224    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920227    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920229    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920232    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920235    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 03:59:53.920693 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920237    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920240    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920243    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920246    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920248    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920251    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920253    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920256    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920258    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920260    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920263    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920266    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920268    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920271    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920274    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920277    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920279    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920282    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920284    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920287    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 03:59:53.921255 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920289    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920292    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920294    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920303    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920305    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920308    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920310    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920313    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920316    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920318    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920321    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920323    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920327    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920331    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920334    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920337    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920339    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920343    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920345    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 03:59:53.921738 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920348    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920352    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920355    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920358    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920360    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920363    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920366    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920370    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920373    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920375    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920378    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920381    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920385    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920388    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920390    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920393    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920395    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920398    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920400    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 03:59:53.922200 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920403    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920405    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920408    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920411    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920413    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920415    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920418    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920420    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920423    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920426    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920428    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920431    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920434    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920436    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920439    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920441    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920444    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920446    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920448    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920451    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 03:59:53.922678 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920826    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920831    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920834    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920836    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920840    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920842    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920845    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920847    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920850    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920853    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920855    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920858    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920860    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920863    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920866    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920868    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920871    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920873    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920876    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920878    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 03:59:53.923156 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920881    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920885    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920888    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920891    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920894    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920897    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920899    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920902    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920905    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920907    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920910    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920913    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920915    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920918    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920921    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920924    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920926    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920929    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920931    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920933    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 03:59:53.923636 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920936    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920939    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920941    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920944    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920946    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920949    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920951    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920953    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920956    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920959    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920963    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920966    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920969    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920972    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920975    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920978    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920981    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920983    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920986    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 03:59:53.924114 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920989    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920991    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920994    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920996    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.920999    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921003    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921006    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921009    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921011    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921014    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921016    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921019    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921022    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921025    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921028    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921031    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921033    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921036    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921039    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921041    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 03:59:53.924683 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921044    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921046    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921048    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921051    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921053    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921056    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921058    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921060    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921063    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921066    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921068    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921070    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921073    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921076    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921078    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921081    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921084    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921086    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921088    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921091    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 03:59:53.925222 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921163    2558 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921174    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921180    2558 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921184    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921192    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921195    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921199    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921203    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921220    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921223    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921227    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921230    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921233    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921236    2558 flags.go:64] FLAG: --cgroup-root=""
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921239    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921242    2558 flags.go:64] FLAG: --client-ca-file=""
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921245    2558 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921247    2558 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921252    2558 flags.go:64] FLAG: --cluster-domain=""
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921255    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921258    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921262    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921265    2558 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921269    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 03:59:53.925894 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921272    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921276    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921279    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921283    2558 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921286    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921289    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921292    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921295    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921300    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921303    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921306    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921309    2558 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921312    2558 flags.go:64] FLAG: --enable-server="true"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921316    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921321    2558 flags.go:64] FLAG: --event-burst="100"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921324    2558 flags.go:64] FLAG: --event-qps="50"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921327    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921330    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921333    2558 flags.go:64] FLAG: --eviction-hard=""
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921336    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921340    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921342    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921348    2558 flags.go:64] FLAG: --eviction-soft=""
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921351    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921354    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 03:59:53.926623 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921357    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921359    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921363    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921366    2558 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921368    2558 flags.go:64] FLAG: --feature-gates=""
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921372    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921375    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921378    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921381    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921384    2558 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921387    2558 flags.go:64] FLAG: --help="false"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921390    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-142-5.ec2.internal"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921393    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921396    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921399    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921402    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921405    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921408    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921411    2558 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921414    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921417    2558 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921420    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921423    2558 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921426    2558 flags.go:64] FLAG: --kube-reserved=""
Jul 09 03:59:53.927764 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921429    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921432    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921435    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921437    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921440    2558 flags.go:64] FLAG: --lock-file=""
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921444    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921447    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921450    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921455    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921458    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921461    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921464    2558 flags.go:64] FLAG: --logging-format="text"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921466    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921469    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921472    2558 flags.go:64] FLAG: --manifest-url=""
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921475    2558 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921480    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921483    2558 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921487    2558 flags.go:64] FLAG: --max-pods="110"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921490    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921493    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921496    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921499    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921502    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921505    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 03:59:53.928741 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921508    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921515    2558 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921518    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921521    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921524    2558 flags.go:64] FLAG: --pod-cidr=""
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921527    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921532    2558 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921535    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921538    2558 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921541    2558 flags.go:64] FLAG: --port="10250"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921544    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921547    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04aeb96bcfa4a1664"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921550    2558 flags.go:64] FLAG: --qos-reserved=""
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921554    2558 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921557    2558 flags.go:64] FLAG: --register-node="true"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921560    2558 flags.go:64] FLAG: --register-with-taints=""
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921563    2558 flags.go:64] FLAG: --registry-burst="10"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921566    2558 flags.go:64] FLAG: --registry-qps="5"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921569    2558 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921572    2558 flags.go:64] FLAG: --reserved-memory=""
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921575    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921578    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921581    2558 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921584    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921587    2558 flags.go:64] FLAG: --runonce="false"
Jul 09 03:59:53.929599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921590    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921593    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921596    2558 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921598    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921601    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921604    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921607    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921610    2558 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921615    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921618    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921621    2558 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921624    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921627    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921630    2558 flags.go:64] FLAG: --system-cgroups=""
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921633    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921638    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921641    2558 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921644    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921651    2558 flags.go:64] FLAG: --tls-min-version=""
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921653    2558 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921657    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921660    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921663    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921666    2558 flags.go:64] FLAG: --v="2"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921670    2558 flags.go:64] FLAG: --version="false"
Jul 09 03:59:53.930201 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921674    2558 flags.go:64] FLAG: --vmodule=""
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921678    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.921681    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921795    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921800    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921804    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921808    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921811    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921814    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921817    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921820    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921823    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921826    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921828    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921831    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921835    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921838    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921840    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921843    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 03:59:53.930843 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921846    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921848    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921851    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921853    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921856    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921859    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921861    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921864    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921868    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921871    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921873    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921876    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921878    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921881    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921884    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921886    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921889    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921892    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921894    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921897    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 03:59:53.931323 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921900    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921902    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921905    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921908    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921911    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921914    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921916    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921919    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921922    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921925    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921928    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921930    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921933    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921935    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921937    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921940    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921943    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921945    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921948    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 03:59:53.931939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921950    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921955    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921957    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921960    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921962    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921964    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921967    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921970    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921972    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921974    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921977    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921979    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921982    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921986    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921989    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921992    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921995    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.921998    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922000    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 03:59:53.932705 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922003    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922005    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922009    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922012    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922014    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922017    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922019    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922022    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922024    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922027    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922029    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922032    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922035    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922037    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922041    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922043    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922046    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922048    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922051    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922053    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 03:59:53.933296 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922056    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922058    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922061    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922063    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.922066    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.923054    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.930842    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.930857    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.930874    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.930884    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930942    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930946    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930950    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930952    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 03:59:53.933779 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930957    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930960    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930963    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930966    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930969    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930973    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930976    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930979    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930982    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930985    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930988    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930990    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930993    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930996    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.930998    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931001    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931003    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931006    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931008    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931011    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 03:59:53.934121 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931013    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931016    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931019    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931021    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931024    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931027    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931030    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931033    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931035    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931039    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931043    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931046    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931049    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931052    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931055    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931058    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931061    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931064    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931066    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 03:59:53.934613 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931068    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931071    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931073    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931076    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931078    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931081    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931083    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931086    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931089    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931091    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931094    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931096    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931099    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931102    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931105    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931107    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931110    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931113    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931116    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931119    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 03:59:53.935125 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931121    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931124    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931126    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931129    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931131    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931134    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931136    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931139    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931142    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931144    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931147    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931149    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931153    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931157    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931159    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931162    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931165    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931167    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931170    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 03:59:53.935615 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931173    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931175    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931178    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931180    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931182    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931185    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931188    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931190    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931192    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931195    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931197    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931200    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931202    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931217    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931220    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931223    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 03:59:53.936177 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931226    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.931231    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931344    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931350    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931353    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931355    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931358    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931361    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931364    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931367    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931369    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931372    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931374    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931377    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931380    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 03:59:53.936596 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931382    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931385    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931387    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931391    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931395    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931398    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931401    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931403    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931406    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931408    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931411    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931413    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931415    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931418    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931420    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931423    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931426    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931429    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931431    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931434    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 03:59:53.936994 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931437    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931439    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931441    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931445    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931448    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931451    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931454    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931457    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931460    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931462    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931465    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931468    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931470    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931473    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931475    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931478    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931480    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931483    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931485    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 03:59:53.937750 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931487    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931490    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931492    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931495    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931497    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931499    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931502    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931504    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931508    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931511    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931514    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931517    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931520    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931523    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931525    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931528    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931530    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931532    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931535    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 03:59:53.938546 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931537    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931540    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931542    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931545    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931547    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931549    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931552    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931554    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931557    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931560    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931562    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931565    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931567    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931570    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931572    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931575    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931577    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931580    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931583    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 03:59:53.939065 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931585    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931588    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931590    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931592    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931639    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931689    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931695    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931699    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:53.931704    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.931710    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.932539    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.934598    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.935790    2558 server.go:1019] "Starting client certificate rotation"
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.935902    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 03:59:53.939727 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.935945    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 03:59:53.940160 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.938329    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 03:59:53.941171 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.941156    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 03:59:53.948604 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.948584    2558 log.go:25] "Validated CRI v1 runtime API"
Jul 09 03:59:53.951930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.951915    2558 log.go:25] "Validated CRI v1 image API"
Jul 09 03:59:53.952765 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.952750    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 03:59:53.956398 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.956376    2558 fs.go:135] Filesystem UUIDs: map[1bb55660-316e-4b3f-a4a6-16e861c26b21:/dev/nvme0n1p4 43d437d6-9bee-4711-809b-498651872fe5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 09 03:59:53.956471 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.956397    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 03:59:53.961584 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.961565    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 03:59:53.963524 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.963399    2558 manager.go:216] Machine: {Timestamp:2026-07-09 03:59:53.959997686 +0000 UTC m=+0.335001495 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100216 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27515ddb1ae6f1eba6e2947427aada SystemUUID:ec27515d-db1a-e6f1-eba6-e2947427aada BootID:5aca890f-edb1-4ea4-bb5b-550f52efda5a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f8:b2:f6:70:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f8:b2:f6:70:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:6c:3c:4b:12:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 03:59:53.963524 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.963518    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 03:59:53.963671 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.963638    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 03:59:53.967986 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.967964    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 03:59:53.968121 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.967988    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-5.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 03:59:53.969296 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.969286    2558 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 03:59:53.969332 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.969299    2558 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 03:59:53.969393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.969306    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 03:59:53.970610 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.970568    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 03:59:53.971983 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.971969    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 03:59:53.976638 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.976625    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 03:59:53.976750 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.976740    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 03:59:53.976783 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.976771    2558 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 03:59:53.976783 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.976781    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 03:59:53.976854 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.976791    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 03:59:53.976854 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.976800    2558 kubelet.go:398] "Adding apiserver pod source"
Jul 09 03:59:53.976854 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.976809    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 03:59:53.977259 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.977247    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 03:59:53.977303 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.977268    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 03:59:53.980114 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.980099    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 09 03:59:53.981874 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.981856    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsmmf"
Jul 09 03:59:53.981991 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.981978    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 03:59:53.982036 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.982008    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 03:59:53.984193 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984182    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 03:59:53.984246 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984199    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 03:59:53.984246 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984216    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 03:59:53.984246 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984221    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 03:59:53.984246 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984231    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 03:59:53.984246 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984237    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 03:59:53.984246 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984245    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 03:59:53.984393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984250    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 03:59:53.984393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984256    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 03:59:53.984393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984261    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 03:59:53.984393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984269    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 03:59:53.984393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.984275    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 03:59:53.985009 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.985001    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 03:59:53.985042 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.985009    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 03:59:53.987770 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.987741    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 03:59:53.987854 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:53.987787    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 03:59:53.987854 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:53.987785    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 03:59:53.988338 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.988202    2558 server.go:1268] "Started kubelet"
Jul 09 03:59:53.988338 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.988284    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 03:59:53.988448 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.988385    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 03:59:53.988487 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.988453    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 03:59:53.988986 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.988967    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsmmf"
Jul 09 03:59:53.989165 ip-10-0-142-5 systemd[1]: Started Kubernetes Kubelet.
Jul 09 03:59:53.990232 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.990203    2558 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 03:59:53.990577 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.990561    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 03:59:53.993753 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:53.992999    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 03:59:53.994014 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:53.993015    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-5.ec2.internal.18c0831891da520a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-5.ec2.internal,UID:ip-10-0-142-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-5.ec2.internal,},FirstTimestamp:2026-07-09 03:59:53.988174346 +0000 UTC m=+0.363178156,LastTimestamp:2026-07-09 03:59:53.988174346 +0000 UTC m=+0.363178156,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-5.ec2.internal,}"
Jul 09 03:59:53.998673 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.998650    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 03:59:53.998673 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.998654    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 03:59:53.998806 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:53.998744    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:53.999235 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.999221    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 03:59:53.999235 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.999237    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 03:59:53.999338 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.999221    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 03:59:53.999393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.999343    2558 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 03:59:53.999393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.999349    2558 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 03:59:53.999672 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.999655    2558 factory.go:55] Registering systemd factory
Jul 09 03:59:53.999672 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.999675    2558 factory.go:223] Registration of the systemd container factory successfully
Jul 09 03:59:53.999942 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.999925    2558 factory.go:153] Registering CRI-O factory
Jul 09 03:59:53.999942 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:53.999942    2558 factory.go:223] Registration of the crio container factory successfully
Jul 09 03:59:54.000646 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.000247    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 03:59:54.000646 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.000282    2558 factory.go:103] Registering Raw factory
Jul 09 03:59:54.000646 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.000301    2558 manager.go:1196] Started watching for new ooms in manager
Jul 09 03:59:54.001014 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.000994    2558 manager.go:349] Starting recovery of all containers
Jul 09 03:59:54.002081 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.002060    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 03:59:54.004974 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.004932    2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-5.ec2.internal\" not found" node="ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.012104 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.012088    2558 manager.go:354] Recovery completed
Jul 09 03:59:54.016866 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.016853    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 03:59:54.021276 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.021251    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 03:59:54.021345 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.021291    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 03:59:54.021345 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.021306    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientPID"
Jul 09 03:59:54.021951 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.021935    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 03:59:54.021951 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.021945    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 03:59:54.021951 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.021950    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 03:59:54.022123 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.022003    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 03:59:54.023176 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.023161    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 03:59:54.023260 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.023190    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 03:59:54.023260 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.023221    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientPID"
Jul 09 03:59:54.023360 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.023336    2558 policy_none.go:49] "None policy: Start"
Jul 09 03:59:54.023360 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.023353    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 03:59:54.023446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.023363    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 03:59:54.024425 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.024411    2558 policy_none.go:47] "Start"
Jul 09 03:59:54.047948 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.047792    2558 manager.go:337] "Starting Device Plugin manager"
Jul 09 03:59:54.048065 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.047969    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 03:59:54.048065 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.047982    2558 server.go:87] "Starting device plugin registration server"
Jul 09 03:59:54.048256 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.048241    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 03:59:54.048354 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.048257    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 03:59:54.048453 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.048440    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 03:59:54.048539 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.048529    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 03:59:54.048595 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.048539    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 03:59:54.049015 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.048996    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 03:59:54.049096 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.049049    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.102092 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.102009    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 03:59:54.103287 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.103269    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 03:59:54.103393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.103308    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 03:59:54.103393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.103330    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 03:59:54.103393 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.103366    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 03:59:54.106263 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.106238    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 03:59:54.148394 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.148365    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 03:59:54.149428 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.149411    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 03:59:54.149522 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.149441    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 03:59:54.149522 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.149452    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientPID"
Jul 09 03:59:54.149522 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.149473    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.160923 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.160901    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.161012 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.160925    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-5.ec2.internal\": node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.174265 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.174237    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.203584 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.203550    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal"]
Jul 09 03:59:54.203723 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.203631    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 03:59:54.204620 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.204601    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 03:59:54.204691 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.204640    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 03:59:54.204691 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.204654    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientPID"
Jul 09 03:59:54.205920 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.205905    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 03:59:54.206013 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.206001    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.206074 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.206033    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 03:59:54.206750 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.206731    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 03:59:54.206844 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.206731    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 03:59:54.206844 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.206788    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 03:59:54.206844 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.206762    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 03:59:54.206844 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.206805    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientPID"
Jul 09 03:59:54.206844 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.206832    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientPID"
Jul 09 03:59:54.208016 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.208004    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.208078 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.208026    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 03:59:54.208720 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.208706    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 03:59:54.208800 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.208732    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 03:59:54.208800 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.208753    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeHasSufficientPID"
Jul 09 03:59:54.223754 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.223734    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-5.ec2.internal\" not found" node="ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.227654 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.227640    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-5.ec2.internal\" not found" node="ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.275131 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.275101    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.301397 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.301365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c494349d68c4b7f96a25ffa7a5ab288a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal\" (UID: \"c494349d68c4b7f96a25ffa7a5ab288a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.301529 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.301401    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c494349d68c4b7f96a25ffa7a5ab288a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal\" (UID: \"c494349d68c4b7f96a25ffa7a5ab288a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.301529 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.301425    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cff16d302c0c6fa07ce53eea472441e3-config\") pod \"kube-apiserver-proxy-ip-10-0-142-5.ec2.internal\" (UID: \"cff16d302c0c6fa07ce53eea472441e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.375703 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.375617    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.401998 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.401967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c494349d68c4b7f96a25ffa7a5ab288a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal\" (UID: \"c494349d68c4b7f96a25ffa7a5ab288a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.401998 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.401990    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c494349d68c4b7f96a25ffa7a5ab288a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal\" (UID: \"c494349d68c4b7f96a25ffa7a5ab288a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.402161 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.402017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c494349d68c4b7f96a25ffa7a5ab288a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal\" (UID: \"c494349d68c4b7f96a25ffa7a5ab288a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.402161 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.402035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cff16d302c0c6fa07ce53eea472441e3-config\") pod \"kube-apiserver-proxy-ip-10-0-142-5.ec2.internal\" (UID: \"cff16d302c0c6fa07ce53eea472441e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.402161 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.402061    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cff16d302c0c6fa07ce53eea472441e3-config\") pod \"kube-apiserver-proxy-ip-10-0-142-5.ec2.internal\" (UID: \"cff16d302c0c6fa07ce53eea472441e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.402161 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.402087    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c494349d68c4b7f96a25ffa7a5ab288a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal\" (UID: \"c494349d68c4b7f96a25ffa7a5ab288a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.476288 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.476243    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.526870 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.526832    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.531898 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.531226    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:54.576410 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.576383    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.676934 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.676864    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.777435 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.777405    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.877772 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.877733    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.937167 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.937075    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 03:59:54.937799 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.937265    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 03:59:54.937799 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.937265    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 03:59:54.978766 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:54.978739    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:54.995810 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.995779    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 03:54:54 +0000 UTC" deadline="2028-03-23 00:06:34.8854734 +0000 UTC"
Jul 09 03:59:54.995925 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:54.995825    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14948h6m39.889652799s"
Jul 09 03:59:55.078924 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:55.078876    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:55.114573 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:55.114522    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff16d302c0c6fa07ce53eea472441e3.slice/crio-4564614d97f05c2121014f7c095f978123d89e08981b33bf7cbaacc574e7d308 WatchSource:0}: Error finding container 4564614d97f05c2121014f7c095f978123d89e08981b33bf7cbaacc574e7d308: Status 404 returned error can't find the container with id 4564614d97f05c2121014f7c095f978123d89e08981b33bf7cbaacc574e7d308
Jul 09 03:59:55.114973 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:55.114944    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc494349d68c4b7f96a25ffa7a5ab288a.slice/crio-b8e8810efb77596551fe3688526e3f8c16b6718e57ff889186df9bfa9e1ab4e8 WatchSource:0}: Error finding container b8e8810efb77596551fe3688526e3f8c16b6718e57ff889186df9bfa9e1ab4e8: Status 404 returned error can't find the container with id b8e8810efb77596551fe3688526e3f8c16b6718e57ff889186df9bfa9e1ab4e8
Jul 09 03:59:55.118889 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.118869    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 03:59:55.179380 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:55.179339    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:55.277796 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.277725    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 03:59:55.280148 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:55.280129    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:55.385644 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:55.385612    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:55.486079 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:55.486037    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-5.ec2.internal\" not found"
Jul 09 03:59:55.522713 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.522683    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 03:59:55.599546 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.599256    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:55.607418 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.607365    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 03:59:55.609131 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.609102    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"
Jul 09 03:59:55.619889 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.619861    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 03:59:55.967831 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.967732    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 03:59:55.977969 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.977945    2558 apiserver.go:52] "Watching apiserver"
Jul 09 03:59:55.989167 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.989134    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 03:59:55.990599 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.990569    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-6rnsk","openshift-multus/multus-additional-cni-plugins-xbcdx","openshift-ovn-kubernetes/ovnkube-node-hg587","kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck","openshift-cluster-node-tuning-operator/tuned-kff9v","openshift-image-registry/node-ca-vvqj2","openshift-multus/network-metrics-daemon-h7zph","openshift-network-diagnostics/network-check-target-lrg9n","openshift-network-operator/iptables-alerter-qlcqv","kube-system/global-pull-secret-syncer-4f6rf","kube-system/konnectivity-agent-gjl8b","openshift-dns/node-resolver-h6v76","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal"]
Jul 09 03:59:55.992742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.992580    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:55.994353 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.994098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:55.995729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.995475    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 03:59:55.995729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.995509    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-85vm9\""
Jul 09 03:59:55.995729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.995559    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 03:59:55.995933 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.995790    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 03:59:55.996631 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.996599    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:55.997235 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.997123    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 03:59:55.997235 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.997138    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nssbm\""
Jul 09 03:59:55.997235 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.997199    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 03:59:55.997631 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.997530    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 03:59:55.998770 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.998434    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:55.998770 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:55.998531    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 03:59:55.999512 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.999489    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 03:59:55.999823 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.999697    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 03:59:55.999823 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.999740    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 03:59:56.000611 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.000251    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4lwdd\""
Jul 09 03:59:56.000611 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.000528    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 03:59:56.002152 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.000805    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.002152 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.000872    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 03:59:56.002152 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.001316    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 03:59:56.002152 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:55.999697    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 03:59:56.004937 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.004892    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 03:59:56.005037 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.004972    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 03:59:56.005037 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.004979    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rcfwm\""
Jul 09 03:59:56.005167 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.005150    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.005221 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.005212    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 03:59:56.006422 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.006402    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.007853 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.007832    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 03:59:56.007962 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.007912    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w6fg4\""
Jul 09 03:59:56.008035 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.007970    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 03:59:56.008684 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.008616    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 03:59:56.008819 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.008796    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 03:59:56.008921 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.008883    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 03:59:56.008921 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.008914    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-86lh2\""
Jul 09 03:59:56.009600 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.009580    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 03:59:56.009698 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.009648    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 03:59:56.009798 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.009773    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 03:59:56.010672 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.010601    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.011692 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.011651    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-run-ovn-kubernetes\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.011692 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.011685    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0578c1a-c573-4cbd-9d2f-9f30f0954753-ovnkube-script-lib\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.011866 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.011712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-sys-fs\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.011866 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.011741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxqwj\" (UniqueName: \"kubernetes.io/projected/c07f69f0-be28-45f5-8268-0d6ec240ae5c-kube-api-access-qxqwj\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.011866 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.011808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6tv\" (UniqueName: \"kubernetes.io/projected/29ca9faa-1fb7-4223-a246-f01696f726e4-kube-api-access-7b6tv\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:56.011866 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.011833    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szngc\" (UniqueName: \"kubernetes.io/projected/d5d7965a-ab05-4758-997b-883c2e788152-kube-api-access-szngc\") pod \"node-ca-vvqj2\" (UID: \"d5d7965a-ab05-4758-997b-883c2e788152\") " pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.012076 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.011876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-node-log\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012076 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.011968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-kubelet\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012076 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.011992    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-run-openvswitch\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012076 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012016    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.012076 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012042    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bcebba85-9e95-4579-8f4d-ebc426873d55-host-slash\") pod \"iptables-alerter-qlcqv\" (UID: \"bcebba85-9e95-4579-8f4d-ebc426873d55\") " pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.012076 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-log-socket\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012127    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-slash\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012196    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5d7965a-ab05-4758-997b-883c2e788152-serviceca\") pod \"node-ca-vvqj2\" (UID: \"d5d7965a-ab05-4758-997b-883c2e788152\") " pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012220    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-socket-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012246    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-registration-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-run-netns\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012327    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0578c1a-c573-4cbd-9d2f-9f30f0954753-env-overrides\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012352    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5d7965a-ab05-4758-997b-883c2e788152-host\") pod \"node-ca-vvqj2\" (UID: \"d5d7965a-ab05-4758-997b-883c2e788152\") " pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwz9\" (UniqueName: \"kubernetes.io/projected/c0578c1a-c573-4cbd-9d2f-9f30f0954753-kube-api-access-hwwz9\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012419 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012399    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-run-ovn\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012436    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012440    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-systemd-units\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012486    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-var-lib-openvswitch\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.012499    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012516    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-etc-openvswitch\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012542    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0578c1a-c573-4cbd-9d2f-9f30f0954753-ovnkube-config\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012615    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-device-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-cni-netd\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012716    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0578c1a-c573-4cbd-9d2f-9f30f0954753-ovn-node-metrics-cert\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bcebba85-9e95-4579-8f4d-ebc426873d55-iptables-alerter-script\") pod \"iptables-alerter-qlcqv\" (UID: \"bcebba85-9e95-4579-8f4d-ebc426873d55\") " pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012782    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4r5x\" (UniqueName: \"kubernetes.io/projected/bcebba85-9e95-4579-8f4d-ebc426873d55-kube-api-access-l4r5x\") pod \"iptables-alerter-qlcqv\" (UID: \"bcebba85-9e95-4579-8f4d-ebc426873d55\") " pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-run-systemd\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012833    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-cni-bin\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.012930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.012936    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.013685 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.013253    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 03:59:56.013685 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.013276    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7l2fp\""
Jul 09 03:59:56.013685 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.013322    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 03:59:56.014467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.014446    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 03:59:56.016424 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.016399    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.016985 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.016965    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 03:59:56.017437 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.017403    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zn7lw\""
Jul 09 03:59:56.017539 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.017506    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 03:59:56.017993 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.017972    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 03:59:56.018757 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.018739    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 03:59:56.019252 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.019072    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f95pl\""
Jul 09 03:59:56.020397 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.020354    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 03:59:56.035407 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.035379    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hjb7"
Jul 09 03:59:56.044611 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.044587    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hjb7"
Jul 09 03:59:56.100989 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.100959    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 03:59:56.109192 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.109135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal" event={"ID":"cff16d302c0c6fa07ce53eea472441e3","Type":"ContainerStarted","Data":"4564614d97f05c2121014f7c095f978123d89e08981b33bf7cbaacc574e7d308"}
Jul 09 03:59:56.111523 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.111494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal" event={"ID":"c494349d68c4b7f96a25ffa7a5ab288a","Type":"ContainerStarted","Data":"b8e8810efb77596551fe3688526e3f8c16b6718e57ff889186df9bfa9e1ab4e8"}
Jul 09 03:59:56.113727 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-run-systemd\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.113848 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-cni-bin\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.113848 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.113848 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113794    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-kubernetes\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.113848 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113821    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2bc32fee-2e66-4598-866b-2a22de571626-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.113848 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113845    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94d7622f-22d9-4229-b344-78fd728ed2d7-agent-certs\") pod \"konnectivity-agent-gjl8b\" (UID: \"94d7622f-22d9-4229-b344-78fd728ed2d7\") " pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113854    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-cni-bin\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113865    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113840    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-run-systemd\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113869    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94d7622f-22d9-4229-b344-78fd728ed2d7-konnectivity-ca\") pod \"konnectivity-agent-gjl8b\" (UID: \"94d7622f-22d9-4229-b344-78fd728ed2d7\") " pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6tv\" (UniqueName: \"kubernetes.io/projected/29ca9faa-1fb7-4223-a246-f01696f726e4-kube-api-access-7b6tv\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72e5e84b-cb3b-4a19-a75c-a1150fd26a50-hosts-file\") pod \"node-resolver-h6v76\" (UID: \"72e5e84b-cb3b-4a19-a75c-a1150fd26a50\") " pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.113989    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-sysctl-d\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114015    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-sysctl-conf\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2bc32fee-2e66-4598-866b-2a22de571626-cni-binary-copy\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114081    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-systemd\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.114179 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114146    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5645ae66-7358-4067-b40c-3c0d5322f0f0-cni-binary-copy\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.114207    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114268    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-run-multus-certs\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.114328    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs podName:29ca9faa-1fb7-4223-a246-f01696f726e4 nodeName:}" failed. No retries permitted until 2026-07-09 03:59:56.614290539 +0000 UTC m=+2.989294338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs") pod "network-metrics-daemon-h7zph" (UID: "29ca9faa-1fb7-4223-a246-f01696f726e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114356    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc32fee-2e66-4598-866b-2a22de571626-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0578c1a-c573-4cbd-9d2f-9f30f0954753-env-overrides\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5d7965a-ab05-4758-997b-883c2e788152-host\") pod \"node-ca-vvqj2\" (UID: \"d5d7965a-ab05-4758-997b-883c2e788152\") " pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114443    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-tuned\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114459    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-cnibin\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwz9\" (UniqueName: \"kubernetes.io/projected/c0578c1a-c573-4cbd-9d2f-9f30f0954753-kube-api-access-hwwz9\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114491    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-systemd-units\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114524    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-etc-openvswitch\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114549    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0578c1a-c573-4cbd-9d2f-9f30f0954753-ovnkube-config\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114554    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5d7965a-ab05-4758-997b-883c2e788152-host\") pod \"node-ca-vvqj2\" (UID: \"d5d7965a-ab05-4758-997b-883c2e788152\") " pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114573    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bcebba85-9e95-4579-8f4d-ebc426873d55-iptables-alerter-script\") pod \"iptables-alerter-qlcqv\" (UID: \"bcebba85-9e95-4579-8f4d-ebc426873d55\") " pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114612    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-etc-openvswitch\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.114729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114739    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114623    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4r5x\" (UniqueName: \"kubernetes.io/projected/bcebba85-9e95-4579-8f4d-ebc426873d55-kube-api-access-l4r5x\") pod \"iptables-alerter-qlcqv\" (UID: \"bcebba85-9e95-4579-8f4d-ebc426873d55\") " pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-cnibin\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114783    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-systemd-units\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114812    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-host\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114837    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-os-release\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114861    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28wm2\" (UniqueName: \"kubernetes.io/projected/5645ae66-7358-4067-b40c-3c0d5322f0f0-kube-api-access-28wm2\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114887    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114916    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ced5b3ab-d85d-457c-890b-26f1e50b6059-kubelet-config\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114941    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1418801-6e37-4925-bf7f-82de59a7ac06-tmp\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-var-lib-cni-multus\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114985    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0578c1a-c573-4cbd-9d2f-9f30f0954753-env-overrides\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.114991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-os-release\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115038    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-run-ovn-kubernetes\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0578c1a-c573-4cbd-9d2f-9f30f0954753-ovnkube-script-lib\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115099    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-sys-fs\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxqwj\" (UniqueName: \"kubernetes.io/projected/c07f69f0-be28-45f5-8268-0d6ec240ae5c-kube-api-access-qxqwj\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.115687 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115116    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-run-ovn-kubernetes\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115128    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0578c1a-c573-4cbd-9d2f-9f30f0954753-ovnkube-config\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115134    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-hostroot\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115146    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bcebba85-9e95-4579-8f4d-ebc426873d55-iptables-alerter-script\") pod \"iptables-alerter-qlcqv\" (UID: \"bcebba85-9e95-4579-8f4d-ebc426873d55\") " pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115181    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mn5k\" (UniqueName: \"kubernetes.io/projected/72e5e84b-cb3b-4a19-a75c-a1150fd26a50-kube-api-access-9mn5k\") pod \"node-resolver-h6v76\" (UID: \"72e5e84b-cb3b-4a19-a75c-a1150fd26a50\") " pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115184    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-sys-fs\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szngc\" (UniqueName: \"kubernetes.io/projected/d5d7965a-ab05-4758-997b-883c2e788152-kube-api-access-szngc\") pod \"node-ca-vvqj2\" (UID: \"d5d7965a-ab05-4758-997b-883c2e788152\") " pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-node-log\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115291    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-kubelet\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115318    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-run-openvswitch\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115344    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-conf-dir\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115366    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-node-log\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115391    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-kubelet\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115396    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vts\" (UniqueName: \"kubernetes.io/projected/2bc32fee-2e66-4598-866b-2a22de571626-kube-api-access-h8vts\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115424    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e5e84b-cb3b-4a19-a75c-a1150fd26a50-tmp-dir\") pod \"node-resolver-h6v76\" (UID: \"72e5e84b-cb3b-4a19-a75c-a1150fd26a50\") " pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115433    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-run-openvswitch\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.116467 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115445    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115461    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bcebba85-9e95-4579-8f4d-ebc426873d55-host-slash\") pod \"iptables-alerter-qlcqv\" (UID: \"bcebba85-9e95-4579-8f4d-ebc426873d55\") " pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115497    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-modprobe-d\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115524    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bcebba85-9e95-4579-8f4d-ebc426873d55-host-slash\") pod \"iptables-alerter-qlcqv\" (UID: \"bcebba85-9e95-4579-8f4d-ebc426873d55\") " pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115555    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-run\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-lib-modules\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115609    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-system-cni-dir\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115620    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0578c1a-c573-4cbd-9d2f-9f30f0954753-ovnkube-script-lib\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115653    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-socket-dir-parent\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115679    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-system-cni-dir\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115705    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115739    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-log-socket\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115776    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-slash\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5d7965a-ab05-4758-997b-883c2e788152-serviceca\") pod \"node-ca-vvqj2\" (UID: \"d5d7965a-ab05-4758-997b-883c2e788152\") " pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115826    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-log-socket\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115878    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-slash\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.115900    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-socket-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.117116 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-registration-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-var-lib-kubelet\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116072    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-registration-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-cni-dir\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116096    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-socket-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116112    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-daemon-config\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116143    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-run-netns\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-sysconfig\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-sys\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116213    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-run-k8s-cni-cncf-io\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116245    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-run-netns\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116252    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-etc-kubernetes\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116281    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5d7965a-ab05-4758-997b-883c2e788152-serviceca\") pod \"node-ca-vvqj2\" (UID: \"d5d7965a-ab05-4758-997b-883c2e788152\") " pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116318    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116348    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ced5b3ab-d85d-457c-890b-26f1e50b6059-dbus\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-run-ovn\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116408    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-var-lib-openvswitch\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.117729 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116432    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-device-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116447    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-run-ovn\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116458    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbsw\" (UniqueName: \"kubernetes.io/projected/e1418801-6e37-4925-bf7f-82de59a7ac06-kube-api-access-fjbsw\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116490    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-var-lib-openvswitch\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116493    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-run-netns\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116526    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c07f69f0-be28-45f5-8268-0d6ec240ae5c-device-dir\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-cni-netd\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116559    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0578c1a-c573-4cbd-9d2f-9f30f0954753-ovn-node-metrics-cert\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116588    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-var-lib-cni-bin\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116596    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0578c1a-c573-4cbd-9d2f-9f30f0954753-host-cni-netd\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116614    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-var-lib-kubelet\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.118279 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.116945    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 03:59:56.120705 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.120685    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0578c1a-c573-4cbd-9d2f-9f30f0954753-ovn-node-metrics-cert\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.127002 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.126977    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6tv\" (UniqueName: \"kubernetes.io/projected/29ca9faa-1fb7-4223-a246-f01696f726e4-kube-api-access-7b6tv\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:56.127121 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.127035    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwz9\" (UniqueName: \"kubernetes.io/projected/c0578c1a-c573-4cbd-9d2f-9f30f0954753-kube-api-access-hwwz9\") pod \"ovnkube-node-hg587\" (UID: \"c0578c1a-c573-4cbd-9d2f-9f30f0954753\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.127557 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.127519    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szngc\" (UniqueName: \"kubernetes.io/projected/d5d7965a-ab05-4758-997b-883c2e788152-kube-api-access-szngc\") pod \"node-ca-vvqj2\" (UID: \"d5d7965a-ab05-4758-997b-883c2e788152\") " pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.127860 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.127837    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxqwj\" (UniqueName: \"kubernetes.io/projected/c07f69f0-be28-45f5-8268-0d6ec240ae5c-kube-api-access-qxqwj\") pod \"aws-ebs-csi-driver-node-w9hck\" (UID: \"c07f69f0-be28-45f5-8268-0d6ec240ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.128393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.128376    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4r5x\" (UniqueName: \"kubernetes.io/projected/bcebba85-9e95-4579-8f4d-ebc426873d55-kube-api-access-l4r5x\") pod \"iptables-alerter-qlcqv\" (UID: \"bcebba85-9e95-4579-8f4d-ebc426873d55\") " pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.216968 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.216912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-hostroot\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.216968 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.216965    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mn5k\" (UniqueName: \"kubernetes.io/projected/72e5e84b-cb3b-4a19-a75c-a1150fd26a50-kube-api-access-9mn5k\") pod \"node-resolver-h6v76\" (UID: \"72e5e84b-cb3b-4a19-a75c-a1150fd26a50\") " pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-conf-dir\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217025    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vts\" (UniqueName: \"kubernetes.io/projected/2bc32fee-2e66-4598-866b-2a22de571626-kube-api-access-h8vts\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217043    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-hostroot\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217054    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e5e84b-cb3b-4a19-a75c-a1150fd26a50-tmp-dir\") pod \"node-resolver-h6v76\" (UID: \"72e5e84b-cb3b-4a19-a75c-a1150fd26a50\") " pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217116    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-modprobe-d\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217132    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-conf-dir\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-run\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-lib-modules\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217210    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-system-cni-dir\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-socket-dir-parent\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-system-cni-dir\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217280    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-modprobe-d\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.217306 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217315    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-run\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217337    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-var-lib-kubelet\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217350    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-lib-modules\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-cni-dir\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217374    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-system-cni-dir\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217340    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-socket-dir-parent\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217377    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-system-cni-dir\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-daemon-config\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-sysconfig\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217423    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-var-lib-kubelet\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.217429    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217457    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e5e84b-cb3b-4a19-a75c-a1150fd26a50-tmp-dir\") pod \"node-resolver-h6v76\" (UID: \"72e5e84b-cb3b-4a19-a75c-a1150fd26a50\") " pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217478    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-sys\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.217496    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret podName:ced5b3ab-d85d-457c-890b-26f1e50b6059 nodeName:}" failed. No retries permitted until 2026-07-09 03:59:56.717474714 +0000 UTC m=+3.092478530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret") pod "global-pull-secret-syncer-4f6rf" (UID: "ced5b3ab-d85d-457c-890b-26f1e50b6059") : object "kube-system"/"original-pull-secret" not registered
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-sys\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217513    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-cni-dir\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217522    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-sysconfig\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217542    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-run-k8s-cni-cncf-io\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.217988 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217582    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-etc-kubernetes\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217610    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217631    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-run-k8s-cni-cncf-io\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217636    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-etc-kubernetes\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ced5b3ab-d85d-457c-890b-26f1e50b6059-dbus\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217694    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbsw\" (UniqueName: \"kubernetes.io/projected/e1418801-6e37-4925-bf7f-82de59a7ac06-kube-api-access-fjbsw\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217720    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-run-netns\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217823    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-var-lib-cni-bin\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217858    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-var-lib-kubelet\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217869    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ced5b3ab-d85d-457c-890b-26f1e50b6059-dbus\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217921    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-var-lib-cni-bin\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217944    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-kubernetes\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217950    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-var-lib-kubelet\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217888    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-kubernetes\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217975    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-run-netns\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217980    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5645ae66-7358-4067-b40c-3c0d5322f0f0-multus-daemon-config\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.217991    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2bc32fee-2e66-4598-866b-2a22de571626-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218024    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94d7622f-22d9-4229-b344-78fd728ed2d7-agent-certs\") pod \"konnectivity-agent-gjl8b\" (UID: \"94d7622f-22d9-4229-b344-78fd728ed2d7\") " pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 03:59:56.218875 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94d7622f-22d9-4229-b344-78fd728ed2d7-konnectivity-ca\") pod \"konnectivity-agent-gjl8b\" (UID: \"94d7622f-22d9-4229-b344-78fd728ed2d7\") " pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218082    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72e5e84b-cb3b-4a19-a75c-a1150fd26a50-hosts-file\") pod \"node-resolver-h6v76\" (UID: \"72e5e84b-cb3b-4a19-a75c-a1150fd26a50\") " pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218120    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-sysctl-d\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218138    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72e5e84b-cb3b-4a19-a75c-a1150fd26a50-hosts-file\") pod \"node-resolver-h6v76\" (UID: \"72e5e84b-cb3b-4a19-a75c-a1150fd26a50\") " pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218152    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-sysctl-conf\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218220    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2bc32fee-2e66-4598-866b-2a22de571626-cni-binary-copy\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218270    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-systemd\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218297    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5645ae66-7358-4067-b40c-3c0d5322f0f0-cni-binary-copy\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218323    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-run-multus-certs\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc32fee-2e66-4598-866b-2a22de571626-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218391    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-run-multus-certs\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218392    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-tuned\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218436    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-cnibin\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218476    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-cnibin\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218502    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-host\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-os-release\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218557    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28wm2\" (UniqueName: \"kubernetes.io/projected/5645ae66-7358-4067-b40c-3c0d5322f0f0-kube-api-access-28wm2\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.219742 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218609    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ced5b3ab-d85d-457c-890b-26f1e50b6059-kubelet-config\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1418801-6e37-4925-bf7f-82de59a7ac06-tmp\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-var-lib-cni-multus\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-os-release\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218710    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2bc32fee-2e66-4598-866b-2a22de571626-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218764    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94d7622f-22d9-4229-b344-78fd728ed2d7-konnectivity-ca\") pod \"konnectivity-agent-gjl8b\" (UID: \"94d7622f-22d9-4229-b344-78fd728ed2d7\") " pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218778    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-cnibin\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218778    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-os-release\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218855    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2bc32fee-2e66-4598-866b-2a22de571626-cni-binary-copy\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218888    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-sysctl-d\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218910    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2bc32fee-2e66-4598-866b-2a22de571626-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218937    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-host\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218940    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ced5b3ab-d85d-457c-890b-26f1e50b6059-kubelet-config\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218947    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5645ae66-7358-4067-b40c-3c0d5322f0f0-cni-binary-copy\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218975    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-cnibin\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218981    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-host-var-lib-cni-multus\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.218355    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-sysctl-conf\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.220446 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.219004    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5645ae66-7358-4067-b40c-3c0d5322f0f0-os-release\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.221236 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.219031    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-systemd\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.221236 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.219445    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc32fee-2e66-4598-866b-2a22de571626-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.221236 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.221126    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1418801-6e37-4925-bf7f-82de59a7ac06-tmp\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.221359 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.221321    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1418801-6e37-4925-bf7f-82de59a7ac06-etc-tuned\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.221393 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.221377    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94d7622f-22d9-4229-b344-78fd728ed2d7-agent-certs\") pod \"konnectivity-agent-gjl8b\" (UID: \"94d7622f-22d9-4229-b344-78fd728ed2d7\") " pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 03:59:56.227839 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.227811    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mn5k\" (UniqueName: \"kubernetes.io/projected/72e5e84b-cb3b-4a19-a75c-a1150fd26a50-kube-api-access-9mn5k\") pod \"node-resolver-h6v76\" (UID: \"72e5e84b-cb3b-4a19-a75c-a1150fd26a50\") " pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.229437 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.229413    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 03:59:56.229550 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.229445    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 03:59:56.229550 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.229457    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pwlfj for pod openshift-network-diagnostics/network-check-target-lrg9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 03:59:56.232554 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.229903    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj podName:973a9365-b1ed-41b0-9fbb-4f7c621ccc7a nodeName:}" failed. No retries permitted until 2026-07-09 03:59:56.729877601 +0000 UTC m=+3.104881412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pwlfj" (UniqueName: "kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj") pod "network-check-target-lrg9n" (UID: "973a9365-b1ed-41b0-9fbb-4f7c621ccc7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 03:59:56.233585 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.233560    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vts\" (UniqueName: \"kubernetes.io/projected/2bc32fee-2e66-4598-866b-2a22de571626-kube-api-access-h8vts\") pod \"multus-additional-cni-plugins-xbcdx\" (UID: \"2bc32fee-2e66-4598-866b-2a22de571626\") " pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.233700 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.233560    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbsw\" (UniqueName: \"kubernetes.io/projected/e1418801-6e37-4925-bf7f-82de59a7ac06-kube-api-access-fjbsw\") pod \"tuned-kff9v\" (UID: \"e1418801-6e37-4925-bf7f-82de59a7ac06\") " pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.234680 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.234656    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28wm2\" (UniqueName: \"kubernetes.io/projected/5645ae66-7358-4067-b40c-3c0d5322f0f0-kube-api-access-28wm2\") pod \"multus-6rnsk\" (UID: \"5645ae66-7358-4067-b40c-3c0d5322f0f0\") " pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.306675 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.306640    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck"
Jul 09 03:59:56.316930 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.316896    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qlcqv"
Jul 09 03:59:56.327987 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.327955    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 03:59:56.333937 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.333916    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vvqj2"
Jul 09 03:59:56.342731 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.342694    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kff9v"
Jul 09 03:59:56.349910 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.349884    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6rnsk"
Jul 09 03:59:56.359373 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.359348    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xbcdx"
Jul 09 03:59:56.367311 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.367280    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 03:59:56.373092 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.373064    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6v76"
Jul 09 03:59:56.390924 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.390888    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 03:59:56.621889 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.621792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:56.622072 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.621955    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 03:59:56.622072 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.622047    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs podName:29ca9faa-1fb7-4223-a246-f01696f726e4 nodeName:}" failed. No retries permitted until 2026-07-09 03:59:57.62202657 +0000 UTC m=+3.997030371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs") pod "network-metrics-daemon-h7zph" (UID: "29ca9faa-1fb7-4223-a246-f01696f726e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 03:59:56.722879 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.722847    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:56.723029 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.722964    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 03:59:56.723029 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.723016    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret podName:ced5b3ab-d85d-457c-890b-26f1e50b6059 nodeName:}" failed. No retries permitted until 2026-07-09 03:59:57.723002056 +0000 UTC m=+4.098005854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret") pod "global-pull-secret-syncer-4f6rf" (UID: "ced5b3ab-d85d-457c-890b-26f1e50b6059") : object "kube-system"/"original-pull-secret" not registered
Jul 09 03:59:56.823933 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:56.823901    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 03:59:56.824095 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.824043    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 03:59:56.824095 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.824059    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 03:59:56.824095 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.824068    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pwlfj for pod openshift-network-diagnostics/network-check-target-lrg9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 03:59:56.824199 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:56.824119    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj podName:973a9365-b1ed-41b0-9fbb-4f7c621ccc7a nodeName:}" failed. No retries permitted until 2026-07-09 03:59:57.824104747 +0000 UTC m=+4.199108547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwlfj" (UniqueName: "kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj") pod "network-check-target-lrg9n" (UID: "973a9365-b1ed-41b0-9fbb-4f7c621ccc7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 03:59:57.047428 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:57.047303    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 03:54:56 +0000 UTC" deadline="2028-01-17 08:53:12.202202315 +0000 UTC"
Jul 09 03:59:57.047428 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:57.047372    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13372h53m15.154833475s"
Jul 09 03:59:57.103805 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:57.103774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 03:59:57.104090 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.103881    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 03:59:57.104090 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:57.103901    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:57.104090 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.104049    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 03:59:57.115017 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:57.114989    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc32fee_2e66_4598_866b_2a22de571626.slice/crio-e446cfdec05ba0788e67c1b806dbd245ae6b89dc3734d98e661810bb376863f3 WatchSource:0}: Error finding container e446cfdec05ba0788e67c1b806dbd245ae6b89dc3734d98e661810bb376863f3: Status 404 returned error can't find the container with id e446cfdec05ba0788e67c1b806dbd245ae6b89dc3734d98e661810bb376863f3
Jul 09 03:59:57.129850 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:57.129818    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07f69f0_be28_45f5_8268_0d6ec240ae5c.slice/crio-61e61de6219cac4de3dde905d43dcadfb31d464b8d969c8d0842be2459270ef7 WatchSource:0}: Error finding container 61e61de6219cac4de3dde905d43dcadfb31d464b8d969c8d0842be2459270ef7: Status 404 returned error can't find the container with id 61e61de6219cac4de3dde905d43dcadfb31d464b8d969c8d0842be2459270ef7
Jul 09 03:59:57.134939 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:57.134902    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d7965a_ab05_4758_997b_883c2e788152.slice/crio-27833207c41b6a027f5af4c10d437ead3a82b126d8d0c3bc17c550d629fc7036 WatchSource:0}: Error finding container 27833207c41b6a027f5af4c10d437ead3a82b126d8d0c3bc17c550d629fc7036: Status 404 returned error can't find the container with id 27833207c41b6a027f5af4c10d437ead3a82b126d8d0c3bc17c550d629fc7036
Jul 09 03:59:57.137818 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:57.137753    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0578c1a_c573_4cbd_9d2f_9f30f0954753.slice/crio-d7fe68b59070633891c12694f6651b1751b9fd1ee526a9e7e66683f1ea421ad7 WatchSource:0}: Error finding container d7fe68b59070633891c12694f6651b1751b9fd1ee526a9e7e66683f1ea421ad7: Status 404 returned error can't find the container with id d7fe68b59070633891c12694f6651b1751b9fd1ee526a9e7e66683f1ea421ad7
Jul 09 03:59:57.138570 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:57.138545    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e5e84b_cb3b_4a19_a75c_a1150fd26a50.slice/crio-c9e9694ebae352084a8d5df00f03d0d30dc2243850042c029ff31ed0bac8bd67 WatchSource:0}: Error finding container c9e9694ebae352084a8d5df00f03d0d30dc2243850042c029ff31ed0bac8bd67: Status 404 returned error can't find the container with id c9e9694ebae352084a8d5df00f03d0d30dc2243850042c029ff31ed0bac8bd67
Jul 09 03:59:57.139894 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:57.139721    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5645ae66_7358_4067_b40c_3c0d5322f0f0.slice/crio-23fe938475b2eb0d24737930d42bc8bb5e1a6335e3a36a64eaaaeb178d4295c7 WatchSource:0}: Error finding container 23fe938475b2eb0d24737930d42bc8bb5e1a6335e3a36a64eaaaeb178d4295c7: Status 404 returned error can't find the container with id 23fe938475b2eb0d24737930d42bc8bb5e1a6335e3a36a64eaaaeb178d4295c7
Jul 09 03:59:57.140536 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:57.140383    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1418801_6e37_4925_bf7f_82de59a7ac06.slice/crio-e52de833f1119d91847481a03c29b8811bbf8dfb42218b596c56486ad315780f WatchSource:0}: Error finding container e52de833f1119d91847481a03c29b8811bbf8dfb42218b596c56486ad315780f: Status 404 returned error can't find the container with id e52de833f1119d91847481a03c29b8811bbf8dfb42218b596c56486ad315780f
Jul 09 03:59:57.142951 ip-10-0-142-5 kubenswrapper[2558]: W0709 03:59:57.142823    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d7622f_22d9_4229_b344_78fd728ed2d7.slice/crio-7ac64a7e05f67dbbf987c81330c8898ffab9f124a39bf4938124a662b4e4bbd9 WatchSource:0}: Error finding container 7ac64a7e05f67dbbf987c81330c8898ffab9f124a39bf4938124a662b4e4bbd9: Status 404 returned error can't find the container with id 7ac64a7e05f67dbbf987c81330c8898ffab9f124a39bf4938124a662b4e4bbd9
Jul 09 03:59:57.630952 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:57.629882    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:57.630952 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.630376    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 03:59:57.630952 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.630445    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs podName:29ca9faa-1fb7-4223-a246-f01696f726e4 nodeName:}" failed. No retries permitted until 2026-07-09 03:59:59.630422637 +0000 UTC m=+6.005426437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs") pod "network-metrics-daemon-h7zph" (UID: "29ca9faa-1fb7-4223-a246-f01696f726e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 03:59:57.730621 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:57.730585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:57.730897 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.730783    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 03:59:57.730897 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.730846    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret podName:ced5b3ab-d85d-457c-890b-26f1e50b6059 nodeName:}" failed. No retries permitted until 2026-07-09 03:59:59.730825734 +0000 UTC m=+6.105829537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret") pod "global-pull-secret-syncer-4f6rf" (UID: "ced5b3ab-d85d-457c-890b-26f1e50b6059") : object "kube-system"/"original-pull-secret" not registered
Jul 09 03:59:57.831314 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:57.831256    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 03:59:57.831502 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.831472    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 03:59:57.831502 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.831493    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 03:59:57.831626 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.831507    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pwlfj for pod openshift-network-diagnostics/network-check-target-lrg9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 03:59:57.831626 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:57.831567    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj podName:973a9365-b1ed-41b0-9fbb-4f7c621ccc7a nodeName:}" failed. No retries permitted until 2026-07-09 03:59:59.831547423 +0000 UTC m=+6.206551224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwlfj" (UniqueName: "kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj") pod "network-check-target-lrg9n" (UID: "973a9365-b1ed-41b0-9fbb-4f7c621ccc7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 03:59:58.048121 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.048021    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 03:54:56 +0000 UTC" deadline="2028-02-13 18:25:51.849363226 +0000 UTC"
Jul 09 03:59:58.048121 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.048068    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14030h25m53.801300021s"
Jul 09 03:59:58.104539 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.104504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:58.104705 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:58.104639    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 03:59:58.138644 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.138574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal" event={"ID":"cff16d302c0c6fa07ce53eea472441e3","Type":"ContainerStarted","Data":"813ba59c268a2b851d7f5556115ffea40ecccf5e6a299335abb146a747e89093"}
Jul 09 03:59:58.146697 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.146528    2558 generic.go:356] "Generic (PLEG): container finished" podID="c494349d68c4b7f96a25ffa7a5ab288a" containerID="454ca649c01ee9d6536a03b16d1ad40c111a3ce1c78c821eb411b0c7664eda9b" exitCode=0
Jul 09 03:59:58.146697 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.146638    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal" event={"ID":"c494349d68c4b7f96a25ffa7a5ab288a","Type":"ContainerDied","Data":"454ca649c01ee9d6536a03b16d1ad40c111a3ce1c78c821eb411b0c7664eda9b"}
Jul 09 03:59:58.153842 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.153773    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qlcqv" event={"ID":"bcebba85-9e95-4579-8f4d-ebc426873d55","Type":"ContainerStarted","Data":"129fd0be61b9f3ec49e9e28c95770a35d3d9ac4692e37374c25d2347c0825c27"}
Jul 09 03:59:58.163646 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.163571    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6rnsk" event={"ID":"5645ae66-7358-4067-b40c-3c0d5322f0f0","Type":"ContainerStarted","Data":"23fe938475b2eb0d24737930d42bc8bb5e1a6335e3a36a64eaaaeb178d4295c7"}
Jul 09 03:59:58.178909 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.172154    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.ec2.internal" podStartSLOduration=2.172133276 podStartE2EDuration="2.172133276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:59:58.155452391 +0000 UTC m=+4.530456214" watchObservedRunningTime="2026-07-09 03:59:58.172133276 +0000 UTC m=+4.547137098"
Jul 09 03:59:58.184723 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.179637    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kff9v" event={"ID":"e1418801-6e37-4925-bf7f-82de59a7ac06","Type":"ContainerStarted","Data":"e52de833f1119d91847481a03c29b8811bbf8dfb42218b596c56486ad315780f"}
Jul 09 03:59:58.204207 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.204136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" event={"ID":"c0578c1a-c573-4cbd-9d2f-9f30f0954753","Type":"ContainerStarted","Data":"d7fe68b59070633891c12694f6651b1751b9fd1ee526a9e7e66683f1ea421ad7"}
Jul 09 03:59:58.217932 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.217880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbcdx" event={"ID":"2bc32fee-2e66-4598-866b-2a22de571626","Type":"ContainerStarted","Data":"e446cfdec05ba0788e67c1b806dbd245ae6b89dc3734d98e661810bb376863f3"}
Jul 09 03:59:58.233900 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.233862    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gjl8b" event={"ID":"94d7622f-22d9-4229-b344-78fd728ed2d7","Type":"ContainerStarted","Data":"7ac64a7e05f67dbbf987c81330c8898ffab9f124a39bf4938124a662b4e4bbd9"}
Jul 09 03:59:58.246918 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.246866    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6v76" event={"ID":"72e5e84b-cb3b-4a19-a75c-a1150fd26a50","Type":"ContainerStarted","Data":"c9e9694ebae352084a8d5df00f03d0d30dc2243850042c029ff31ed0bac8bd67"}
Jul 09 03:59:58.253273 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.253240    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vvqj2" event={"ID":"d5d7965a-ab05-4758-997b-883c2e788152","Type":"ContainerStarted","Data":"27833207c41b6a027f5af4c10d437ead3a82b126d8d0c3bc17c550d629fc7036"}
Jul 09 03:59:58.269328 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:58.269277    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck" event={"ID":"c07f69f0-be28-45f5-8268-0d6ec240ae5c","Type":"ContainerStarted","Data":"61e61de6219cac4de3dde905d43dcadfb31d464b8d969c8d0842be2459270ef7"}
Jul 09 03:59:59.104178 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:59.104094    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 03:59:59.104672 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.104227    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 03:59:59.104672 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:59.104654    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:59.104791 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.104767    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 03:59:59.294578 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:59.294535    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal" event={"ID":"c494349d68c4b7f96a25ffa7a5ab288a","Type":"ContainerStarted","Data":"617e55c8c47f993bc0613e5c080962664c2ef1d6ffd07fe24ce3c5b7bde68ee6"}
Jul 09 03:59:59.312744 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:59.311683    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.ec2.internal" podStartSLOduration=3.311661802 podStartE2EDuration="3.311661802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 03:59:59.31133216 +0000 UTC m=+5.686336003" watchObservedRunningTime="2026-07-09 03:59:59.311661802 +0000 UTC m=+5.686665626"
Jul 09 03:59:59.650730 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:59.650679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 03:59:59.650938 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.650909    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 03:59:59.651004 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.650981    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs podName:29ca9faa-1fb7-4223-a246-f01696f726e4 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:03.650958463 +0000 UTC m=+10.025962266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs") pod "network-metrics-daemon-h7zph" (UID: "29ca9faa-1fb7-4223-a246-f01696f726e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 03:59:59.751163 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:59.751105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 03:59:59.751382 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.751338    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 03:59:59.751452 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.751408    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret podName:ced5b3ab-d85d-457c-890b-26f1e50b6059 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:03.75138863 +0000 UTC m=+10.126392434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret") pod "global-pull-secret-syncer-4f6rf" (UID: "ced5b3ab-d85d-457c-890b-26f1e50b6059") : object "kube-system"/"original-pull-secret" not registered
Jul 09 03:59:59.852235 ip-10-0-142-5 kubenswrapper[2558]: I0709 03:59:59.852192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 03:59:59.852433 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.852408    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 03:59:59.852433 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.852432    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 03:59:59.852541 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.852446    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pwlfj for pod openshift-network-diagnostics/network-check-target-lrg9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 03:59:59.852541 ip-10-0-142-5 kubenswrapper[2558]: E0709 03:59:59.852507    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj podName:973a9365-b1ed-41b0-9fbb-4f7c621ccc7a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:03.852488018 +0000 UTC m=+10.227491819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwlfj" (UniqueName: "kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj") pod "network-check-target-lrg9n" (UID: "973a9365-b1ed-41b0-9fbb-4f7c621ccc7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 04:00:00.109643 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:00.109560    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:00.110108 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:00.109696    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:01.104525 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:01.104484    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:01.104710 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:01.104629    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:01.105119 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:01.105096    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:01.105242 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:01.105220    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:02.109030 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:02.109001    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:02.109554 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:02.109134    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:03.104025 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:03.103911    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:03.104025 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:03.103944    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:03.104285 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.104062    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:03.104586 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.104555    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:03.690884 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:03.690845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:03.691348 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.690999    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 04:00:03.691348 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.691073    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs podName:29ca9faa-1fb7-4223-a246-f01696f726e4 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:11.691049923 +0000 UTC m=+18.066053736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs") pod "network-metrics-daemon-h7zph" (UID: "29ca9faa-1fb7-4223-a246-f01696f726e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 04:00:03.792355 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:03.792201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:03.792585 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.792393    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 04:00:03.792585 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.792478    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret podName:ced5b3ab-d85d-457c-890b-26f1e50b6059 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:11.792456083 +0000 UTC m=+18.167459898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret") pod "global-pull-secret-syncer-4f6rf" (UID: "ced5b3ab-d85d-457c-890b-26f1e50b6059") : object "kube-system"/"original-pull-secret" not registered
Jul 09 04:00:03.893028 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:03.892939    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:03.893226 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.893153    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 04:00:03.893226 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.893175    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 04:00:03.893226 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.893188    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pwlfj for pod openshift-network-diagnostics/network-check-target-lrg9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 04:00:03.893424 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:03.893282    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj podName:973a9365-b1ed-41b0-9fbb-4f7c621ccc7a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:11.893244831 +0000 UTC m=+18.268248631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwlfj" (UniqueName: "kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj") pod "network-check-target-lrg9n" (UID: "973a9365-b1ed-41b0-9fbb-4f7c621ccc7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 04:00:04.104716 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:04.104614    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:04.104881 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:04.104740    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:05.104585 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:05.104520    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:05.105096 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:05.104657    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:05.105096 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:05.105003    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:05.105223 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:05.105115    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:06.104458 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:06.104424    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:06.104621 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:06.104560    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:07.104112 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:07.104081    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:07.104307 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:07.104263    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:07.104390 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:07.104080    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:07.104452 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:07.104428    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:08.104351 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:08.104328    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:08.104647 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:08.104433    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:09.103573 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:09.103530    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:09.103764 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:09.103583    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:09.103764 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:09.103686    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:09.103877 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:09.103766    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:10.103604 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:10.103571    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:10.104061 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:10.103694    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:11.104568 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:11.104521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:11.105112 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:11.104521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:11.105112 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.104676    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:11.105112 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.104749    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:11.747347 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:11.747315    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:11.747507 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.747420    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 04:00:11.747507 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.747470    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs podName:29ca9faa-1fb7-4223-a246-f01696f726e4 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:27.747456915 +0000 UTC m=+34.122460711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs") pod "network-metrics-daemon-h7zph" (UID: "29ca9faa-1fb7-4223-a246-f01696f726e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 04:00:11.847751 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:11.847720    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:11.847911 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.847870    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 04:00:11.847947 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.847932    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret podName:ced5b3ab-d85d-457c-890b-26f1e50b6059 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:27.847915813 +0000 UTC m=+34.222919610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret") pod "global-pull-secret-syncer-4f6rf" (UID: "ced5b3ab-d85d-457c-890b-26f1e50b6059") : object "kube-system"/"original-pull-secret" not registered
Jul 09 04:00:11.948405 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:11.948366    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:11.948532 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.948484    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 04:00:11.948532 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.948497    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 04:00:11.948532 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.948506    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pwlfj for pod openshift-network-diagnostics/network-check-target-lrg9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 04:00:11.948635 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:11.948554    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj podName:973a9365-b1ed-41b0-9fbb-4f7c621ccc7a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:27.948541307 +0000 UTC m=+34.323545103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwlfj" (UniqueName: "kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj") pod "network-check-target-lrg9n" (UID: "973a9365-b1ed-41b0-9fbb-4f7c621ccc7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 04:00:12.104005 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:12.103924    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:12.104158 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:12.104051    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:13.104280 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:13.104241    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:13.104719 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:13.104251    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:13.104719 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:13.104358    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:13.104719 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:13.104466    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:14.105019 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:14.104983    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:14.105421 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:14.105098    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:15.104386 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:15.104352    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:15.104584 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:15.104354    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:15.104584 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:15.104467    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:15.104584 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:15.104558    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:16.104624 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:16.104577    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:16.105079 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:16.104729    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:17.104609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.104304    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:17.104774 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.104362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:17.104774 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:17.104669    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:17.105388 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:17.104779    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:17.331597 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.331510    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck" event={"ID":"c07f69f0-be28-45f5-8268-0d6ec240ae5c","Type":"ContainerStarted","Data":"9f0061ef8860909deedffb1ab3c95cdae72838748e9f0ffea402ce7d3ca0b295"}
Jul 09 04:00:17.333198 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.333171    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6rnsk" event={"ID":"5645ae66-7358-4067-b40c-3c0d5322f0f0","Type":"ContainerStarted","Data":"656618f984def62e6f96ae2aaeac684986d7c2f0e1f52c9b1db29202d06a98e0"}
Jul 09 04:00:17.334881 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.334856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kff9v" event={"ID":"e1418801-6e37-4925-bf7f-82de59a7ac06","Type":"ContainerStarted","Data":"9ff04e2bba1f5391cdcef73a4f24db6fd5d38635395dff59fe116b2a72ad2423"}
Jul 09 04:00:17.338348 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.338323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" event={"ID":"c0578c1a-c573-4cbd-9d2f-9f30f0954753","Type":"ContainerStarted","Data":"e0263823f1efaf22d58d32e9c05c39629beafdaee060e2b794d01b38934dfd88"}
Jul 09 04:00:17.338451 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.338403    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" event={"ID":"c0578c1a-c573-4cbd-9d2f-9f30f0954753","Type":"ContainerStarted","Data":"0ce23ea4ccbba066fae3efe1e575a2c7637734e28a2e2ee0aaf2eadde19a5062"}
Jul 09 04:00:17.338504 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.338487    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" event={"ID":"c0578c1a-c573-4cbd-9d2f-9f30f0954753","Type":"ContainerStarted","Data":"17ce44b4ebef934700cc21ad0d296397dcbf47bab623baa62656b5cec07bd05c"}
Jul 09 04:00:17.338556 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.338541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" event={"ID":"c0578c1a-c573-4cbd-9d2f-9f30f0954753","Type":"ContainerStarted","Data":"b4d5bce54b38838321a27d1caf9b141664ed1bc733b269214b6d64f7d1a046ea"}
Jul 09 04:00:17.338602 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.338593    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" event={"ID":"c0578c1a-c573-4cbd-9d2f-9f30f0954753","Type":"ContainerStarted","Data":"a3917b98ee933daeda054bfee080f8b364b1956378e9a726e571e524e2e4674a"}
Jul 09 04:00:17.338654 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.338639    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" event={"ID":"c0578c1a-c573-4cbd-9d2f-9f30f0954753","Type":"ContainerStarted","Data":"fb4576af29154f9f69de94e7db5d71dc963d0e2200af8301688319fbaad0f91c"}
Jul 09 04:00:17.339888 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.339860    2558 generic.go:356] "Generic (PLEG): container finished" podID="2bc32fee-2e66-4598-866b-2a22de571626" containerID="c34db1699bacbc0e9758d174c96ef47b6de7d68468a79fc6febf67b9eeb6104f" exitCode=0
Jul 09 04:00:17.340000 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.339936    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbcdx" event={"ID":"2bc32fee-2e66-4598-866b-2a22de571626","Type":"ContainerDied","Data":"c34db1699bacbc0e9758d174c96ef47b6de7d68468a79fc6febf67b9eeb6104f"}
Jul 09 04:00:17.343927 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.342319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gjl8b" event={"ID":"94d7622f-22d9-4229-b344-78fd728ed2d7","Type":"ContainerStarted","Data":"a9000b85ef56ccb12d64fd7df4293d527b30a5b1a6944747497ebaf4a6390959"}
Jul 09 04:00:17.346133 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.346112    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6v76" event={"ID":"72e5e84b-cb3b-4a19-a75c-a1150fd26a50","Type":"ContainerStarted","Data":"464ac20026fac43ec436cf26dfeaa8075cb0237bd159a192368985736222d912"}
Jul 09 04:00:17.347583 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.347561    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vvqj2" event={"ID":"d5d7965a-ab05-4758-997b-883c2e788152","Type":"ContainerStarted","Data":"dee583c10a68242c1cb41037da12502bb70ea946c991cbb28e93e41eed230cda"}
Jul 09 04:00:17.349983 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.349943    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6rnsk" podStartSLOduration=3.1129684 podStartE2EDuration="22.349927609s" totalImagesPullingTime="19.236959209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:17.349635697 +0000 UTC m=+23.724639516" watchObservedRunningTime="2026-07-09 04:00:17.349927609 +0000 UTC m=+23.724931428"
Jul 09 04:00:17.365378 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.365328    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kff9v" podStartSLOduration=3.146779103 podStartE2EDuration="22.365315277s" totalImagesPullingTime="19.218536174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:17.364847394 +0000 UTC m=+23.739851213" watchObservedRunningTime="2026-07-09 04:00:17.365315277 +0000 UTC m=+23.740319095"
Jul 09 04:00:17.414115 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.414066    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vvqj2" podStartSLOduration=3.214800664 podStartE2EDuration="22.414051015s" totalImagesPullingTime="19.199250351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:17.413618512 +0000 UTC m=+23.788622331" watchObservedRunningTime="2026-07-09 04:00:17.414051015 +0000 UTC m=+23.789054833"
Jul 09 04:00:17.432999 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.432939    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gjl8b" podStartSLOduration=11.453340929 podStartE2EDuration="22.432923127s" totalImagesPullingTime="10.979582198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:17.432839024 +0000 UTC m=+23.807842843" watchObservedRunningTime="2026-07-09 04:00:17.432923127 +0000 UTC m=+23.807926946"
Jul 09 04:00:17.449784 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.449733    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h6v76" podStartSLOduration=3.252919084 podStartE2EDuration="22.449716687s" totalImagesPullingTime="19.196797603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:17.449241161 +0000 UTC m=+23.824244995" watchObservedRunningTime="2026-07-09 04:00:17.449716687 +0000 UTC m=+23.824720505"
Jul 09 04:00:17.993568 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:17.993543    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 04:00:18.061476 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:18.061356    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T04:00:17.993562306Z","UUID":"f6c1746e-e136-4de5-b8c8-ac29653e34a7","Handler":null,"Name":"","Endpoint":""}
Jul 09 04:00:18.062986 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:18.062964    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 04:00:18.063125 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:18.063009    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 04:00:18.103998 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:18.103968    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:18.104166 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:18.104081    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:18.352008 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:18.351883    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qlcqv" event={"ID":"bcebba85-9e95-4579-8f4d-ebc426873d55","Type":"ContainerStarted","Data":"a4d4028809b8194291b53b99afe5aef72de2f701e9572baa7b0c6434d48a01b3"}
Jul 09 04:00:18.354292 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:18.354228    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck" event={"ID":"c07f69f0-be28-45f5-8268-0d6ec240ae5c","Type":"ContainerStarted","Data":"d3c9b83f8f5268fdbc349b0cd4b6ad9c71a6b1f101578ddb817ac0d137a4cb5c"}
Jul 09 04:00:18.368443 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:18.368394    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qlcqv" podStartSLOduration=5.177180267 podStartE2EDuration="24.368375296s" totalImagesPullingTime="19.191195029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:18.367948164 +0000 UTC m=+24.742951981" watchObservedRunningTime="2026-07-09 04:00:18.368375296 +0000 UTC m=+24.743379114"
Jul 09 04:00:19.104620 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:19.104584    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:19.104803 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:19.104637    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:19.104803 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:19.104715    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:19.104915 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:19.104852    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:19.360267 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:19.360039    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" event={"ID":"c0578c1a-c573-4cbd-9d2f-9f30f0954753","Type":"ContainerStarted","Data":"244673f568dcdaf8f9fa644071b9030016fdd5cb048f4114f47dc7ec35232870"}
Jul 09 04:00:19.885373 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:19.885342    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 04:00:19.886040 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:19.886011    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 04:00:20.103867 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:20.103834    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:20.104052 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:20.103976    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:20.365391 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:20.365302    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck" event={"ID":"c07f69f0-be28-45f5-8268-0d6ec240ae5c","Type":"ContainerStarted","Data":"759481e4dc4095a346371918a7c73f01e4b8280e207d2db2797bd15e44f47662"}
Jul 09 04:00:20.365934 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:20.365576    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 04:00:20.366156 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:20.366139    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gjl8b"
Jul 09 04:00:20.382998 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:20.382946    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hck" podStartSLOduration=4.488759102 podStartE2EDuration="26.382932641s" totalImagesPullingTime="21.894173539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:20.382546069 +0000 UTC m=+26.757549916" watchObservedRunningTime="2026-07-09 04:00:20.382932641 +0000 UTC m=+26.757936463"
Jul 09 04:00:21.104244 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:21.104197    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:21.104431 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:21.104197    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:21.104431 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:21.104341    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:21.104431 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:21.104411    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:22.104134 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:22.104100    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:22.104720 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:22.104246    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:22.373264 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:22.373021    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" event={"ID":"c0578c1a-c573-4cbd-9d2f-9f30f0954753","Type":"ContainerStarted","Data":"dd545d326fd5f969a132bc370fbabc5a6b5f05507dbd6df2b6b4f310c7ff3c13"}
Jul 09 04:00:23.104457 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.104379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:23.105004 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.104379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:23.105004 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:23.104513    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:23.105004 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:23.104550    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:23.376903 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.376816    2558 generic.go:356] "Generic (PLEG): container finished" podID="2bc32fee-2e66-4598-866b-2a22de571626" containerID="6f940b4cf81d0b69c2b1bf273b289f3e04523949223e703c556710230e99a186" exitCode=0
Jul 09 04:00:23.376903 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.376873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbcdx" event={"ID":"2bc32fee-2e66-4598-866b-2a22de571626","Type":"ContainerDied","Data":"6f940b4cf81d0b69c2b1bf273b289f3e04523949223e703c556710230e99a186"}
Jul 09 04:00:23.377119 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.377087    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 04:00:23.377167 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.377155    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 04:00:23.377255 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.377242    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 04:00:23.393678 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.393655    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 04:00:23.393792 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.393768    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 04:00:23.442913 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:23.442864    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hg587" podStartSLOduration=10.144377291 podStartE2EDuration="29.442848668s" totalImagesPullingTime="19.298471377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:23.441366389 +0000 UTC m=+29.816370219" watchObservedRunningTime="2026-07-09 04:00:23.442848668 +0000 UTC m=+29.817852485"
Jul 09 04:00:24.108203 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:24.108177    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:24.108711 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:24.108370    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:24.303225 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:24.302960    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4f6rf"]
Jul 09 04:00:24.306992 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:24.306964    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7zph"]
Jul 09 04:00:24.307130 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:24.307097    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:24.307256 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:24.307232    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:24.307802 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:24.307756    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lrg9n"]
Jul 09 04:00:24.307915 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:24.307851    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:24.307973 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:24.307936    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:24.379363 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:24.379286    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:24.379561 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:24.379410    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:25.384244 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:25.384199    2558 generic.go:356] "Generic (PLEG): container finished" podID="2bc32fee-2e66-4598-866b-2a22de571626" containerID="6cc5b21b4f194c4061507371be2f477e26c9e2ba6d3984d52fbdc405845542e2" exitCode=0
Jul 09 04:00:25.384663 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:25.384237    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbcdx" event={"ID":"2bc32fee-2e66-4598-866b-2a22de571626","Type":"ContainerDied","Data":"6cc5b21b4f194c4061507371be2f477e26c9e2ba6d3984d52fbdc405845542e2"}
Jul 09 04:00:26.104582 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:26.104555    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:26.104709 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:26.104559    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:26.104709 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:26.104672    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:26.104797 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:26.104559    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:26.104828 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:26.104793    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:26.104828 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:26.104728    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:26.388800 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:26.388761    2558 generic.go:356] "Generic (PLEG): container finished" podID="2bc32fee-2e66-4598-866b-2a22de571626" containerID="8e94d5066089d239d250385480b8f36779ea580af8a18283a2b47469933c29d7" exitCode=0
Jul 09 04:00:26.389315 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:26.388812    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbcdx" event={"ID":"2bc32fee-2e66-4598-866b-2a22de571626","Type":"ContainerDied","Data":"8e94d5066089d239d250385480b8f36779ea580af8a18283a2b47469933c29d7"}
Jul 09 04:00:27.763179 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:27.763107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:27.763520 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:27.763245    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 04:00:27.763520 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:27.763300    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs podName:29ca9faa-1fb7-4223-a246-f01696f726e4 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:59.763285215 +0000 UTC m=+66.138289010 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs") pod "network-metrics-daemon-h7zph" (UID: "29ca9faa-1fb7-4223-a246-f01696f726e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 04:00:27.863671 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:27.863637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:27.863869 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:27.863777    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 04:00:27.863869 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:27.863837    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret podName:ced5b3ab-d85d-457c-890b-26f1e50b6059 nodeName:}" failed. No retries permitted until 2026-07-09 04:00:59.863819259 +0000 UTC m=+66.238823057 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret") pod "global-pull-secret-syncer-4f6rf" (UID: "ced5b3ab-d85d-457c-890b-26f1e50b6059") : object "kube-system"/"original-pull-secret" not registered
Jul 09 04:00:27.964253 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:27.964202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:27.964418 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:27.964342    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 04:00:27.964418 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:27.964363    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 04:00:27.964418 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:27.964376    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pwlfj for pod openshift-network-diagnostics/network-check-target-lrg9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 04:00:27.964578 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:27.964436    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj podName:973a9365-b1ed-41b0-9fbb-4f7c621ccc7a nodeName:}" failed. No retries permitted until 2026-07-09 04:00:59.964417537 +0000 UTC m=+66.339421334 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwlfj" (UniqueName: "kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj") pod "network-check-target-lrg9n" (UID: "973a9365-b1ed-41b0-9fbb-4f7c621ccc7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 04:00:28.104536 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:28.104433    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:28.104688 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:28.104438    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:28.104688 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:28.104574    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:28.104688 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:28.104574    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrg9n" podUID="973a9365-b1ed-41b0-9fbb-4f7c621ccc7a"
Jul 09 04:00:28.104688 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:28.104660    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4f6rf" podUID="ced5b3ab-d85d-457c-890b-26f1e50b6059"
Jul 09 04:00:28.104859 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:28.104763    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7zph" podUID="29ca9faa-1fb7-4223-a246-f01696f726e4"
Jul 09 04:00:29.373474 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.373447    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-5.ec2.internal" event="NodeReady"
Jul 09 04:00:29.373986 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.373571    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 04:00:29.417064 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.417038    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ndcnh"]
Jul 09 04:00:29.419690 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.419661    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.420403 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.420382    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xqjt6"]
Jul 09 04:00:29.422077 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.422060    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 04:00:29.422154 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.422129    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 04:00:29.422250 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.422232    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wgmf2\""
Jul 09 04:00:29.422821 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.422808    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xqjt6"
Jul 09 04:00:29.425322 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.425291    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 04:00:29.425415 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.425324    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 04:00:29.425462 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.425433    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dfx4h\""
Jul 09 04:00:29.426655 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.426467    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 04:00:29.434284 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.434265    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ndcnh"]
Jul 09 04:00:29.434822 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.434805    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xqjt6"]
Jul 09 04:00:29.575817 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.575776    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da771dc-17ac-4f66-9276-1b2fda78a9dd-tmp-dir\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.575817 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.575822    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88sq4\" (UniqueName: \"kubernetes.io/projected/6da771dc-17ac-4f66-9276-1b2fda78a9dd-kube-api-access-88sq4\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.576039 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.575841    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cc1c1e6-f43e-48f9-8488-9ab9c9da5093-cert\") pod \"ingress-canary-xqjt6\" (UID: \"1cc1c1e6-f43e-48f9-8488-9ab9c9da5093\") " pod="openshift-ingress-canary/ingress-canary-xqjt6"
Jul 09 04:00:29.576039 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.575894    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6da771dc-17ac-4f66-9276-1b2fda78a9dd-config-volume\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.576039 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.575955    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.576039 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.575981    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jfj\" (UniqueName: \"kubernetes.io/projected/1cc1c1e6-f43e-48f9-8488-9ab9c9da5093-kube-api-access-z8jfj\") pod \"ingress-canary-xqjt6\" (UID: \"1cc1c1e6-f43e-48f9-8488-9ab9c9da5093\") " pod="openshift-ingress-canary/ingress-canary-xqjt6"
Jul 09 04:00:29.676658 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.676582    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.676658 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.676620    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jfj\" (UniqueName: \"kubernetes.io/projected/1cc1c1e6-f43e-48f9-8488-9ab9c9da5093-kube-api-access-z8jfj\") pod \"ingress-canary-xqjt6\" (UID: \"1cc1c1e6-f43e-48f9-8488-9ab9c9da5093\") " pod="openshift-ingress-canary/ingress-canary-xqjt6"
Jul 09 04:00:29.676843 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.676715    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da771dc-17ac-4f66-9276-1b2fda78a9dd-tmp-dir\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.676843 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.676742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88sq4\" (UniqueName: \"kubernetes.io/projected/6da771dc-17ac-4f66-9276-1b2fda78a9dd-kube-api-access-88sq4\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.676843 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:29.676760    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 04:00:29.676843 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.676803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cc1c1e6-f43e-48f9-8488-9ab9c9da5093-cert\") pod \"ingress-canary-xqjt6\" (UID: \"1cc1c1e6-f43e-48f9-8488-9ab9c9da5093\") " pod="openshift-ingress-canary/ingress-canary-xqjt6"
Jul 09 04:00:29.676843 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:29.676823    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls podName:6da771dc-17ac-4f66-9276-1b2fda78a9dd nodeName:}" failed. No retries permitted until 2026-07-09 04:00:30.176803147 +0000 UTC m=+36.551806957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls") pod "dns-default-ndcnh" (UID: "6da771dc-17ac-4f66-9276-1b2fda78a9dd") : secret "dns-default-metrics-tls" not found
Jul 09 04:00:29.677074 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.676862    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6da771dc-17ac-4f66-9276-1b2fda78a9dd-config-volume\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.677237 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.677180    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da771dc-17ac-4f66-9276-1b2fda78a9dd-tmp-dir\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.677530 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.677510    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6da771dc-17ac-4f66-9276-1b2fda78a9dd-config-volume\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.681105 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.680983    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cc1c1e6-f43e-48f9-8488-9ab9c9da5093-cert\") pod \"ingress-canary-xqjt6\" (UID: \"1cc1c1e6-f43e-48f9-8488-9ab9c9da5093\") " pod="openshift-ingress-canary/ingress-canary-xqjt6"
Jul 09 04:00:29.687252 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.687229    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88sq4\" (UniqueName: \"kubernetes.io/projected/6da771dc-17ac-4f66-9276-1b2fda78a9dd-kube-api-access-88sq4\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:29.687353 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.687277    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jfj\" (UniqueName: \"kubernetes.io/projected/1cc1c1e6-f43e-48f9-8488-9ab9c9da5093-kube-api-access-z8jfj\") pod \"ingress-canary-xqjt6\" (UID: \"1cc1c1e6-f43e-48f9-8488-9ab9c9da5093\") " pod="openshift-ingress-canary/ingress-canary-xqjt6"
Jul 09 04:00:29.734329 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.734291    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xqjt6"
Jul 09 04:00:29.874127 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:29.874092    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xqjt6"]
Jul 09 04:00:29.878909 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:00:29.878884    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc1c1e6_f43e_48f9_8488_9ab9c9da5093.slice/crio-edadf00853845adb24484bb82aec9dd00351ef0818183356410966d63990d43b WatchSource:0}: Error finding container edadf00853845adb24484bb82aec9dd00351ef0818183356410966d63990d43b: Status 404 returned error can't find the container with id edadf00853845adb24484bb82aec9dd00351ef0818183356410966d63990d43b
Jul 09 04:00:30.104329 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.104297    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:30.104503 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.104422    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:30.104766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.104748    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:30.107066 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.107040    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 04:00:30.107176 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.107080    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 04:00:30.108288 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.108131    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 04:00:30.108288 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.108158    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 04:00:30.108288 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.108165    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-25smd\""
Jul 09 04:00:30.108288 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.108269    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fn82r\""
Jul 09 04:00:30.180740 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.180702    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:30.180882 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:30.180851    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 04:00:30.180917 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:30.180912    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls podName:6da771dc-17ac-4f66-9276-1b2fda78a9dd nodeName:}" failed. No retries permitted until 2026-07-09 04:00:31.180896208 +0000 UTC m=+37.555900017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls") pod "dns-default-ndcnh" (UID: "6da771dc-17ac-4f66-9276-1b2fda78a9dd") : secret "dns-default-metrics-tls" not found
Jul 09 04:00:30.399655 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:30.399575    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xqjt6" event={"ID":"1cc1c1e6-f43e-48f9-8488-9ab9c9da5093","Type":"ContainerStarted","Data":"edadf00853845adb24484bb82aec9dd00351ef0818183356410966d63990d43b"}
Jul 09 04:00:31.187358 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:31.187319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:31.187583 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:31.187459    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 04:00:31.187583 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:31.187530    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls podName:6da771dc-17ac-4f66-9276-1b2fda78a9dd nodeName:}" failed. No retries permitted until 2026-07-09 04:00:33.187509478 +0000 UTC m=+39.562513277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls") pod "dns-default-ndcnh" (UID: "6da771dc-17ac-4f66-9276-1b2fda78a9dd") : secret "dns-default-metrics-tls" not found
Jul 09 04:00:33.203059 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:33.202967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:33.203491 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:33.203143    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 04:00:33.203491 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:33.203236    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls podName:6da771dc-17ac-4f66-9276-1b2fda78a9dd nodeName:}" failed. No retries permitted until 2026-07-09 04:00:37.203200946 +0000 UTC m=+43.578204743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls") pod "dns-default-ndcnh" (UID: "6da771dc-17ac-4f66-9276-1b2fda78a9dd") : secret "dns-default-metrics-tls" not found
Jul 09 04:00:33.409697 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:33.409520    2558 generic.go:356] "Generic (PLEG): container finished" podID="2bc32fee-2e66-4598-866b-2a22de571626" containerID="b2fa6f30195b20818e65317f9d28bc1a95c2463b148a778d518da5c81112e2b2" exitCode=0
Jul 09 04:00:33.409855 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:33.409596    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbcdx" event={"ID":"2bc32fee-2e66-4598-866b-2a22de571626","Type":"ContainerDied","Data":"b2fa6f30195b20818e65317f9d28bc1a95c2463b148a778d518da5c81112e2b2"}
Jul 09 04:00:33.411163 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:33.411137    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xqjt6" event={"ID":"1cc1c1e6-f43e-48f9-8488-9ab9c9da5093","Type":"ContainerStarted","Data":"3caeb9057e6009989f732dff3c6b00a5203585a34ef39f4626ecff723b087626"}
Jul 09 04:00:33.446860 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:33.446805    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xqjt6" podStartSLOduration=1.198149248 podStartE2EDuration="4.446784862s" totalImagesPullingTime="3.248635614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:33.445893024 +0000 UTC m=+39.820896833" watchObservedRunningTime="2026-07-09 04:00:33.446784862 +0000 UTC m=+39.821788682"
Jul 09 04:00:34.416404 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:34.416371    2558 generic.go:356] "Generic (PLEG): container finished" podID="2bc32fee-2e66-4598-866b-2a22de571626" containerID="3cd634db05ef1aec312159bb9d1f9be2320f0f85b53add9471b34213383696bb" exitCode=0
Jul 09 04:00:34.416947 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:34.416452    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbcdx" event={"ID":"2bc32fee-2e66-4598-866b-2a22de571626","Type":"ContainerDied","Data":"3cd634db05ef1aec312159bb9d1f9be2320f0f85b53add9471b34213383696bb"}
Jul 09 04:00:35.421921 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:35.421880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbcdx" event={"ID":"2bc32fee-2e66-4598-866b-2a22de571626","Type":"ContainerStarted","Data":"86232821e8c8f88149e5cb7924f523119191dcc89958c98068fbf0c30ce4dcee"}
Jul 09 04:00:35.445296 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:35.445247    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xbcdx" podStartSLOduration=7.328144023 podStartE2EDuration="40.445231807s" totalImagesPullingTime="33.117087784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:00:35.443657507 +0000 UTC m=+41.818661422" watchObservedRunningTime="2026-07-09 04:00:35.445231807 +0000 UTC m=+41.820235621"
Jul 09 04:00:37.229555 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:37.229511    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:37.229948 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:37.229647    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 04:00:37.229948 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:37.229708    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls podName:6da771dc-17ac-4f66-9276-1b2fda78a9dd nodeName:}" failed. No retries permitted until 2026-07-09 04:00:45.229690626 +0000 UTC m=+51.604694422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls") pod "dns-default-ndcnh" (UID: "6da771dc-17ac-4f66-9276-1b2fda78a9dd") : secret "dns-default-metrics-tls" not found
Jul 09 04:00:45.287487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:45.287447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:00:45.287838 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:45.287597    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 04:00:45.287838 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:45.287659    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls podName:6da771dc-17ac-4f66-9276-1b2fda78a9dd nodeName:}" failed. No retries permitted until 2026-07-09 04:01:01.287642922 +0000 UTC m=+67.662646721 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls") pod "dns-default-ndcnh" (UID: "6da771dc-17ac-4f66-9276-1b2fda78a9dd") : secret "dns-default-metrics-tls" not found
Jul 09 04:00:55.395477 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:55.395450    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg587"
Jul 09 04:00:59.790959 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:59.790922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:00:59.793678 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:59.793660    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 04:00:59.801828 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:59.801811    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 04:00:59.801909 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:00:59.801877    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs podName:29ca9faa-1fb7-4223-a246-f01696f726e4 nodeName:}" failed. No retries permitted until 2026-07-09 04:02:03.801858901 +0000 UTC m=+130.176862698 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs") pod "network-metrics-daemon-h7zph" (UID: "29ca9faa-1fb7-4223-a246-f01696f726e4") : secret "metrics-daemon-secret" not found
Jul 09 04:00:59.892037 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:59.891995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:59.895043 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:59.895019    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 04:00:59.906844 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:59.906823    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ced5b3ab-d85d-457c-890b-26f1e50b6059-original-pull-secret\") pod \"global-pull-secret-syncer-4f6rf\" (UID: \"ced5b3ab-d85d-457c-890b-26f1e50b6059\") " pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:00:59.992522 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:59.992486    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:00:59.995400 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:00:59.995380    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 04:01:00.005906 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:00.005879    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 04:01:00.016224 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:00.016182    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlfj\" (UniqueName: \"kubernetes.io/projected/973a9365-b1ed-41b0-9fbb-4f7c621ccc7a-kube-api-access-pwlfj\") pod \"network-check-target-lrg9n\" (UID: \"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a\") " pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:01:00.116526 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:00.116452    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4f6rf"
Jul 09 04:01:00.125966 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:00.125946    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-25smd\""
Jul 09 04:01:00.134203 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:00.134182    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:01:00.269512 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:00.269484    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4f6rf"]
Jul 09 04:01:00.272760 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:00.272734    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced5b3ab_d85d_457c_890b_26f1e50b6059.slice/crio-5dfe3228636c442d865eca62976a9b02d5ddaa597b7c8f7559cd9ee8a641af34 WatchSource:0}: Error finding container 5dfe3228636c442d865eca62976a9b02d5ddaa597b7c8f7559cd9ee8a641af34: Status 404 returned error can't find the container with id 5dfe3228636c442d865eca62976a9b02d5ddaa597b7c8f7559cd9ee8a641af34
Jul 09 04:01:00.282036 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:00.282010    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lrg9n"]
Jul 09 04:01:00.286372 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:00.286350    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973a9365_b1ed_41b0_9fbb_4f7c621ccc7a.slice/crio-a630f4ec1cca5c8111ce9af6ee457a125ff3c0be246c2e2f6fed5ca45faf860a WatchSource:0}: Error finding container a630f4ec1cca5c8111ce9af6ee457a125ff3c0be246c2e2f6fed5ca45faf860a: Status 404 returned error can't find the container with id a630f4ec1cca5c8111ce9af6ee457a125ff3c0be246c2e2f6fed5ca45faf860a
Jul 09 04:01:00.474635 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:00.474603    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lrg9n" event={"ID":"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a","Type":"ContainerStarted","Data":"a630f4ec1cca5c8111ce9af6ee457a125ff3c0be246c2e2f6fed5ca45faf860a"}
Jul 09 04:01:00.475570 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:00.475544    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4f6rf" event={"ID":"ced5b3ab-d85d-457c-890b-26f1e50b6059","Type":"ContainerStarted","Data":"5dfe3228636c442d865eca62976a9b02d5ddaa597b7c8f7559cd9ee8a641af34"}
Jul 09 04:01:01.300265 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:01.300194    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:01:01.300793 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:01.300332    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 04:01:01.300793 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:01.300400    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls podName:6da771dc-17ac-4f66-9276-1b2fda78a9dd nodeName:}" failed. No retries permitted until 2026-07-09 04:01:33.300382866 +0000 UTC m=+99.675386677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls") pod "dns-default-ndcnh" (UID: "6da771dc-17ac-4f66-9276-1b2fda78a9dd") : secret "dns-default-metrics-tls" not found
Jul 09 04:01:05.489267 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:05.489224    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4f6rf" event={"ID":"ced5b3ab-d85d-457c-890b-26f1e50b6059","Type":"ContainerStarted","Data":"f5e951eabfe785dbb7c5a7e43ce68eaf6beddaaa1483776845306d37e2b73dfd"}
Jul 09 04:01:05.504800 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:05.504749    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4f6rf" podStartSLOduration=66.180879502 podStartE2EDuration="1m10.504735078s" totalImagesPullingTime="4.323855576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 03:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:05.50382999 +0000 UTC m=+71.878833808" watchObservedRunningTime="2026-07-09 04:01:05.504735078 +0000 UTC m=+71.879739253"
Jul 09 04:01:07.495500 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:07.495457    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lrg9n" event={"ID":"973a9365-b1ed-41b0-9fbb-4f7c621ccc7a","Type":"ContainerStarted","Data":"e22211443c4caa3ba0ac549225edf5f749fe8071503a755b83f6a2174dae539e"}
Jul 09 04:01:07.495871 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:07.495605    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:01:07.512147 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:07.512101    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lrg9n" podStartSLOduration=65.819812211 podStartE2EDuration="1m12.512087083s" totalImagesPullingTime="6.692274872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:07.511028658 +0000 UTC m=+73.886032496" watchObservedRunningTime="2026-07-09 04:01:07.512087083 +0000 UTC m=+73.887090910"
Jul 09 04:01:18.608162 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.608045    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx"]
Jul 09 04:01:18.610262 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.610245    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx"
Jul 09 04:01:18.614193 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.614171    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5fd7cc5c49-nblmz"]
Jul 09 04:01:18.614331 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.614317    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tctmc\""
Jul 09 04:01:18.616267 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.616247    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.619257 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.619236    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 09 04:01:18.619370 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.619319    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 09 04:01:18.619471 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.619454    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 09 04:01:18.619603 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.619587    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vlk22\""
Jul 09 04:01:18.619961 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.619942    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 04:01:18.620223 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.620189    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 04:01:18.620365 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.620344    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 09 04:01:18.621028 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.621011    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx"]
Jul 09 04:01:18.634642 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.634616    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5fd7cc5c49-nblmz"]
Jul 09 04:01:18.712110 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.712078    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"]
Jul 09 04:01:18.714057 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.714041    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:18.717572 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.717547    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 09 04:01:18.717572 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.717564    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 09 04:01:18.717773 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.717590    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 09 04:01:18.717773 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.717669    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 09 04:01:18.717773 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.717687    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-b8gng\""
Jul 09 04:01:18.721995 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.721977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.722087 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.722028    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-default-certificate\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.722087 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.722048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbtvg\" (UniqueName: \"kubernetes.io/projected/d8701ca9-e2fd-40bc-b807-eaa331eb18a4-kube-api-access-nbtvg\") pod \"network-check-source-989f8b8dc-6grcx\" (UID: \"d8701ca9-e2fd-40bc-b807-eaa331eb18a4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx"
Jul 09 04:01:18.722087 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.722069    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxq5q\" (UniqueName: \"kubernetes.io/projected/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-kube-api-access-pxq5q\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.722221 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.722093    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.722221 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.722115    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-stats-auth\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.728836 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.728817    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"]
Jul 09 04:01:18.823419 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.823380    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.823609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.823461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqcb9\" (UniqueName: \"kubernetes.io/projected/66835b46-56de-45ff-9bf2-997eb83836f5-kube-api-access-pqcb9\") pod \"kube-storage-version-migrator-operator-846bf668f7-cfhxq\" (UID: \"66835b46-56de-45ff-9bf2-997eb83836f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:18.823609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.823492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66835b46-56de-45ff-9bf2-997eb83836f5-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-cfhxq\" (UID: \"66835b46-56de-45ff-9bf2-997eb83836f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:18.823609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.823516    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-default-certificate\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.823609 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:18.823522    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 04:01:18.823609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.823541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbtvg\" (UniqueName: \"kubernetes.io/projected/d8701ca9-e2fd-40bc-b807-eaa331eb18a4-kube-api-access-nbtvg\") pod \"network-check-source-989f8b8dc-6grcx\" (UID: \"d8701ca9-e2fd-40bc-b807-eaa331eb18a4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx"
Jul 09 04:01:18.823609 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:18.823608    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:19.323580867 +0000 UTC m=+85.698584662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : secret "router-metrics-certs-default" not found
Jul 09 04:01:18.823887 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.823650    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxq5q\" (UniqueName: \"kubernetes.io/projected/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-kube-api-access-pxq5q\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.823887 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.823687    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66835b46-56de-45ff-9bf2-997eb83836f5-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-cfhxq\" (UID: \"66835b46-56de-45ff-9bf2-997eb83836f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:18.823887 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.823731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.823887 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.823777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-stats-auth\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.823887 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:18.823862    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:19.323846091 +0000 UTC m=+85.698849906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : configmap references non-existent config key: service-ca.crt
Jul 09 04:01:18.829343 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.825995    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-default-certificate\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.829343 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.826380    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-stats-auth\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.833644 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.833622    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbtvg\" (UniqueName: \"kubernetes.io/projected/d8701ca9-e2fd-40bc-b807-eaa331eb18a4-kube-api-access-nbtvg\") pod \"network-check-source-989f8b8dc-6grcx\" (UID: \"d8701ca9-e2fd-40bc-b807-eaa331eb18a4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx"
Jul 09 04:01:18.833939 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.833916    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxq5q\" (UniqueName: \"kubernetes.io/projected/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-kube-api-access-pxq5q\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:18.919306 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.919269    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx"
Jul 09 04:01:18.924619 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.924588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqcb9\" (UniqueName: \"kubernetes.io/projected/66835b46-56de-45ff-9bf2-997eb83836f5-kube-api-access-pqcb9\") pod \"kube-storage-version-migrator-operator-846bf668f7-cfhxq\" (UID: \"66835b46-56de-45ff-9bf2-997eb83836f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:18.924738 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.924639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66835b46-56de-45ff-9bf2-997eb83836f5-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-cfhxq\" (UID: \"66835b46-56de-45ff-9bf2-997eb83836f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:18.924738 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.924674    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66835b46-56de-45ff-9bf2-997eb83836f5-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-cfhxq\" (UID: \"66835b46-56de-45ff-9bf2-997eb83836f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:18.925258 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.925234    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66835b46-56de-45ff-9bf2-997eb83836f5-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-cfhxq\" (UID: \"66835b46-56de-45ff-9bf2-997eb83836f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:18.926887 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.926865    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66835b46-56de-45ff-9bf2-997eb83836f5-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-cfhxq\" (UID: \"66835b46-56de-45ff-9bf2-997eb83836f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:18.932379 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:18.932360    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqcb9\" (UniqueName: \"kubernetes.io/projected/66835b46-56de-45ff-9bf2-997eb83836f5-kube-api-access-pqcb9\") pod \"kube-storage-version-migrator-operator-846bf668f7-cfhxq\" (UID: \"66835b46-56de-45ff-9bf2-997eb83836f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:19.022504 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:19.022468    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"
Jul 09 04:01:19.039221 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:19.039181    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx"]
Jul 09 04:01:19.042082 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:19.042054    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8701ca9_e2fd_40bc_b807_eaa331eb18a4.slice/crio-728ff2152294c4a591ceba7faef687d317f3894e75cdcac32015d9d10b96796e WatchSource:0}: Error finding container 728ff2152294c4a591ceba7faef687d317f3894e75cdcac32015d9d10b96796e: Status 404 returned error can't find the container with id 728ff2152294c4a591ceba7faef687d317f3894e75cdcac32015d9d10b96796e
Jul 09 04:01:19.149060 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:19.149007    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq"]
Jul 09 04:01:19.151527 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:19.151497    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66835b46_56de_45ff_9bf2_997eb83836f5.slice/crio-cfaefb94f7f0851de1dcfac4c2c26ec0738bfb23c0735690db4f5e28680a847d WatchSource:0}: Error finding container cfaefb94f7f0851de1dcfac4c2c26ec0738bfb23c0735690db4f5e28680a847d: Status 404 returned error can't find the container with id cfaefb94f7f0851de1dcfac4c2c26ec0738bfb23c0735690db4f5e28680a847d
Jul 09 04:01:19.328276 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:19.328163    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:19.328276 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:19.328225    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:19.328497 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:19.328319    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 04:01:19.328497 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:19.328338    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:20.328320213 +0000 UTC m=+86.703324009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : configmap references non-existent config key: service-ca.crt
Jul 09 04:01:19.328497 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:19.328364    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:20.328355794 +0000 UTC m=+86.703359589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : secret "router-metrics-certs-default" not found
Jul 09 04:01:19.527446 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:19.527408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx" event={"ID":"d8701ca9-e2fd-40bc-b807-eaa331eb18a4","Type":"ContainerStarted","Data":"f7cba1c4cd44913f8da49a273044c57cc0b9391e5e7228dadf9f3a1687011423"}
Jul 09 04:01:19.527640 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:19.527472    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx" event={"ID":"d8701ca9-e2fd-40bc-b807-eaa331eb18a4","Type":"ContainerStarted","Data":"728ff2152294c4a591ceba7faef687d317f3894e75cdcac32015d9d10b96796e"}
Jul 09 04:01:19.528521 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:19.528499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq" event={"ID":"66835b46-56de-45ff-9bf2-997eb83836f5","Type":"ContainerStarted","Data":"cfaefb94f7f0851de1dcfac4c2c26ec0738bfb23c0735690db4f5e28680a847d"}
Jul 09 04:01:19.543867 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:19.543822    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6grcx" podStartSLOduration=1.543806644 podStartE2EDuration="1.543806644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:19.543243659 +0000 UTC m=+85.918247474" watchObservedRunningTime="2026-07-09 04:01:19.543806644 +0000 UTC m=+85.918810440"
Jul 09 04:01:20.336429 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:20.336393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:20.336892 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:20.336452    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:20.336892 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:20.336571    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 04:01:20.336892 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:20.336595    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:22.336573075 +0000 UTC m=+88.711576874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : configmap references non-existent config key: service-ca.crt
Jul 09 04:01:20.336892 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:20.336629    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:22.336613056 +0000 UTC m=+88.711616870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : secret "router-metrics-certs-default" not found
Jul 09 04:01:21.535527 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.535496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq" event={"ID":"66835b46-56de-45ff-9bf2-997eb83836f5","Type":"ContainerStarted","Data":"d1a527e97e89fdd55f9343723d7a9daaa1c3587d2ca8bc4ca0f5bf1c31f72286"}
Jul 09 04:01:21.551092 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.551044    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq" podStartSLOduration=1.283270973 podStartE2EDuration="3.55102828s" totalImagesPullingTime="2.267757307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:21.550128826 +0000 UTC m=+87.925132655" watchObservedRunningTime="2026-07-09 04:01:21.55102828 +0000 UTC m=+87.926032144"
Jul 09 04:01:21.947292 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.947261    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6z7dk"]
Jul 09 04:01:21.949863 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.949843    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:21.954017 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.953990    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 04:01:21.954017 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.954006    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 04:01:21.954161 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.954070    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 04:01:21.955046 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.955028    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9vrhl\""
Jul 09 04:01:21.955128 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.955058    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 04:01:21.970310 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:21.970288    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6z7dk"]
Jul 09 04:01:22.050346 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.050317    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6md\" (UniqueName: \"kubernetes.io/projected/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-api-access-zx6md\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.050509 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.050375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.050509 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.050402    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-crio-socket\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.050509 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.050420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-data-volume\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.050644 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.050502    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.151202 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.151168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6md\" (UniqueName: \"kubernetes.io/projected/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-api-access-zx6md\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.151387 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.151239    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.151387 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.151296    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 04:01:22.151387 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.151344    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:22.651329934 +0000 UTC m=+89.026333730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : configmap "kube-rbac-proxy" not found
Jul 09 04:01:22.151507 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.151407    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-crio-socket\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.151507 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.151444    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-data-volume\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.151507 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.151481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.151606 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.151481    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-crio-socket\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.151795 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.151776    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-data-volume\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.151955 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.151941    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:22.151998 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.151991    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:22.651978473 +0000 UTC m=+89.026982268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:22.159063 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.159040    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6md\" (UniqueName: \"kubernetes.io/projected/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-api-access-zx6md\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.352897 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.352810    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:22.352897 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.352861    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:22.353066 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.352958    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 04:01:22.353066 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.352984    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:26.352965882 +0000 UTC m=+92.727969683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : configmap references non-existent config key: service-ca.crt
Jul 09 04:01:22.353066 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.353035    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:26.353028404 +0000 UTC m=+92.728032201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : secret "router-metrics-certs-default" not found
Jul 09 04:01:22.452056 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.452019    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb"]
Jul 09 04:01:22.454265 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.454245    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb"
Jul 09 04:01:22.457021 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.456999    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 04:01:22.457126 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.457066    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 04:01:22.457373 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.457356    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wr8xk\""
Jul 09 04:01:22.465807 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.465787    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb"]
Jul 09 04:01:22.555142 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.555107    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg79s\" (UniqueName: \"kubernetes.io/projected/c518becb-972d-4285-957d-09084364aee5-kube-api-access-zg79s\") pod \"migrator-d57d58cc9-g9qrb\" (UID: \"c518becb-972d-4285-957d-09084364aee5\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb"
Jul 09 04:01:22.655751 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.655708    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.655910 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.655791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:22.655910 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.655849    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 04:01:22.655910 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.655894    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:22.656022 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.655950    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:23.655927527 +0000 UTC m=+90.030931325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : configmap "kube-rbac-proxy" not found
Jul 09 04:01:22.656022 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:22.655982    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:23.655968299 +0000 UTC m=+90.030972099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:22.656022 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.656014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg79s\" (UniqueName: \"kubernetes.io/projected/c518becb-972d-4285-957d-09084364aee5-kube-api-access-zg79s\") pod \"migrator-d57d58cc9-g9qrb\" (UID: \"c518becb-972d-4285-957d-09084364aee5\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb"
Jul 09 04:01:22.666993 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.666970    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg79s\" (UniqueName: \"kubernetes.io/projected/c518becb-972d-4285-957d-09084364aee5-kube-api-access-zg79s\") pod \"migrator-d57d58cc9-g9qrb\" (UID: \"c518becb-972d-4285-957d-09084364aee5\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb"
Jul 09 04:01:22.763328 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.763294    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb"
Jul 09 04:01:22.880406 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:22.880378    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb"]
Jul 09 04:01:22.883764 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:22.883730    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc518becb_972d_4285_957d_09084364aee5.slice/crio-6a0c177251297004902f859de0edb0a547a816e390c11226b74fba909bf7b7f4 WatchSource:0}: Error finding container 6a0c177251297004902f859de0edb0a547a816e390c11226b74fba909bf7b7f4: Status 404 returned error can't find the container with id 6a0c177251297004902f859de0edb0a547a816e390c11226b74fba909bf7b7f4
Jul 09 04:01:23.541251 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:23.541199    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb" event={"ID":"c518becb-972d-4285-957d-09084364aee5","Type":"ContainerStarted","Data":"6a0c177251297004902f859de0edb0a547a816e390c11226b74fba909bf7b7f4"}
Jul 09 04:01:23.665923 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:23.665884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:23.665923 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:23.665929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:23.666352 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:23.666020    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 04:01:23.666352 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:23.666047    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:23.666352 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:23.666087    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:25.666069699 +0000 UTC m=+92.041073498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : configmap "kube-rbac-proxy" not found
Jul 09 04:01:23.666352 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:23.666101    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:25.666095383 +0000 UTC m=+92.041099179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:23.892495 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:23.892414    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6v76_72e5e84b-cb3b-4a19-a75c-a1150fd26a50/dns-node-resolver/0.log"
Jul 09 04:01:24.546023 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:24.545987    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb" event={"ID":"c518becb-972d-4285-957d-09084364aee5","Type":"ContainerStarted","Data":"f5bb0f36e180e4eda4a22026dca089da23feac45070a558d94d01641266a6c02"}
Jul 09 04:01:24.546172 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:24.546047    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb" event={"ID":"c518becb-972d-4285-957d-09084364aee5","Type":"ContainerStarted","Data":"c8b6df7c75c40327757f40006182430931a661f2b181258905c0bc40187414a8"}
Jul 09 04:01:24.562315 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:24.562269    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-g9qrb" podStartSLOduration=1.471072463 podStartE2EDuration="2.562251543s" totalImagesPullingTime="1.09117908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:24.561927147 +0000 UTC m=+90.936930965" watchObservedRunningTime="2026-07-09 04:01:24.562251543 +0000 UTC m=+90.937255360"
Jul 09 04:01:24.892361 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:24.892284    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vvqj2_d5d7965a-ab05-4758-997b-883c2e788152/node-ca/0.log"
Jul 09 04:01:25.681645 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:25.681605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:25.681849 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:25.681662    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:25.681849 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:25.681759    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 04:01:25.681849 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:25.681769    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:25.681849 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:25.681827    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:29.681810262 +0000 UTC m=+96.056814058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : configmap "kube-rbac-proxy" not found
Jul 09 04:01:25.681849 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:25.681842    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:29.681835558 +0000 UTC m=+96.056839355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:25.891179 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:25.891145    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xqjt6_1cc1c1e6-f43e-48f9-8488-9ab9c9da5093/serve-healthcheck-canary/0.log"
Jul 09 04:01:26.388584 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:26.388549    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:26.388972 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:26.388597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:26.388972 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:26.388710    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 04:01:26.388972 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:26.388728    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:34.388709426 +0000 UTC m=+100.763713221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : configmap references non-existent config key: service-ca.crt
Jul 09 04:01:26.388972 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:26.388794    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:34.388782334 +0000 UTC m=+100.763786130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : secret "router-metrics-certs-default" not found
Jul 09 04:01:29.713891 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:29.713853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:29.714326 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:29.713942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:29.714326 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:29.714011    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 04:01:29.714326 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:29.714026    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:29.714326 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:29.714062    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:37.714048019 +0000 UTC m=+104.089051815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : configmap "kube-rbac-proxy" not found
Jul 09 04:01:29.714326 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:29.714080    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:37.714066651 +0000 UTC m=+104.089070451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:33.344594 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:33.344555    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:01:33.346936 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:33.346913    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da771dc-17ac-4f66-9276-1b2fda78a9dd-metrics-tls\") pod \"dns-default-ndcnh\" (UID: \"6da771dc-17ac-4f66-9276-1b2fda78a9dd\") " pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:01:33.632969 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:33.632882    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wgmf2\""
Jul 09 04:01:33.641053 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:33.641037    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:01:33.790914 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:33.790882    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ndcnh"]
Jul 09 04:01:33.793751 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:33.793718    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da771dc_17ac_4f66_9276_1b2fda78a9dd.slice/crio-bb615df7a183471673699d674d4ff97b2e992a841c470c89d43c45ead217326e WatchSource:0}: Error finding container bb615df7a183471673699d674d4ff97b2e992a841c470c89d43c45ead217326e: Status 404 returned error can't find the container with id bb615df7a183471673699d674d4ff97b2e992a841c470c89d43c45ead217326e
Jul 09 04:01:34.452662 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:34.452624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:34.453145 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:34.452778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:34.453145 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:34.452939    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle podName:f1c2b0f3-4be4-4e34-975f-ced29f667c3d nodeName:}" failed. No retries permitted until 2026-07-09 04:01:50.452917366 +0000 UTC m=+116.827921162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle") pod "router-default-5fd7cc5c49-nblmz" (UID: "f1c2b0f3-4be4-4e34-975f-ced29f667c3d") : configmap references non-existent config key: service-ca.crt
Jul 09 04:01:34.455332 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:34.455307    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-metrics-certs\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:34.585185 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:34.585139    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndcnh" event={"ID":"6da771dc-17ac-4f66-9276-1b2fda78a9dd","Type":"ContainerStarted","Data":"bb615df7a183471673699d674d4ff97b2e992a841c470c89d43c45ead217326e"}
Jul 09 04:01:36.592822 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:36.592789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndcnh" event={"ID":"6da771dc-17ac-4f66-9276-1b2fda78a9dd","Type":"ContainerStarted","Data":"48b32af43f09550de465cb44cf40aef004f25674626f1b25a88e94c599fcc2b0"}
Jul 09 04:01:36.593243 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:36.592839    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndcnh" event={"ID":"6da771dc-17ac-4f66-9276-1b2fda78a9dd","Type":"ContainerStarted","Data":"29af130ca78dd34793aab75d55d4ad4b0a9ecd087319c74454e5c021c73634e6"}
Jul 09 04:01:36.593243 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:36.592878    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:01:36.612943 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:36.612883    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ndcnh" podStartSLOduration=65.737300434 podStartE2EDuration="1m7.612868777s" totalImagesPullingTime="1.875568343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:36.611696785 +0000 UTC m=+102.986700642" watchObservedRunningTime="2026-07-09 04:01:36.612868777 +0000 UTC m=+102.987872597"
Jul 09 04:01:37.777419 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:37.777370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:37.777797 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:37.777466    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:37.777797 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:37.777507    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:37.777797 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:01:37.777570    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls podName:17a511ac-2f5c-4fc2-904c-ffeb76048e4c nodeName:}" failed. No retries permitted until 2026-07-09 04:01:53.777552212 +0000 UTC m=+120.152556008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6z7dk" (UID: "17a511ac-2f5c-4fc2-904c-ffeb76048e4c") : secret "insights-runtime-extractor-tls" not found
Jul 09 04:01:37.777915 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:37.777899    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:38.500687 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:38.500653    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lrg9n"
Jul 09 04:01:46.598196 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:46.598169    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ndcnh"
Jul 09 04:01:49.392084 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.392051    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"]
Jul 09 04:01:49.394047 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.394031    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.396808 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.396788    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 04:01:49.397092 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.397073    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 04:01:49.397194 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.397073    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 04:01:49.398370 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.398355    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 04:01:49.406429 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.406405    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"]
Jul 09 04:01:49.568201 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.568164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0480e53-1328-4671-9168-ca55579c4e53-tmp\") pod \"klusterlet-addon-workmgr-8548dc8745-zjzkp\" (UID: \"e0480e53-1328-4671-9168-ca55579c4e53\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.568201 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.568203    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e0480e53-1328-4671-9168-ca55579c4e53-klusterlet-config\") pod \"klusterlet-addon-workmgr-8548dc8745-zjzkp\" (UID: \"e0480e53-1328-4671-9168-ca55579c4e53\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.568427 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.568270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mb42\" (UniqueName: \"kubernetes.io/projected/e0480e53-1328-4671-9168-ca55579c4e53-kube-api-access-6mb42\") pod \"klusterlet-addon-workmgr-8548dc8745-zjzkp\" (UID: \"e0480e53-1328-4671-9168-ca55579c4e53\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.669582 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.669548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0480e53-1328-4671-9168-ca55579c4e53-tmp\") pod \"klusterlet-addon-workmgr-8548dc8745-zjzkp\" (UID: \"e0480e53-1328-4671-9168-ca55579c4e53\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.669582 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.669581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e0480e53-1328-4671-9168-ca55579c4e53-klusterlet-config\") pod \"klusterlet-addon-workmgr-8548dc8745-zjzkp\" (UID: \"e0480e53-1328-4671-9168-ca55579c4e53\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.669788 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.669605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mb42\" (UniqueName: \"kubernetes.io/projected/e0480e53-1328-4671-9168-ca55579c4e53-kube-api-access-6mb42\") pod \"klusterlet-addon-workmgr-8548dc8745-zjzkp\" (UID: \"e0480e53-1328-4671-9168-ca55579c4e53\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.670006 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.669982    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0480e53-1328-4671-9168-ca55579c4e53-tmp\") pod \"klusterlet-addon-workmgr-8548dc8745-zjzkp\" (UID: \"e0480e53-1328-4671-9168-ca55579c4e53\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.672198 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.672176    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e0480e53-1328-4671-9168-ca55579c4e53-klusterlet-config\") pod \"klusterlet-addon-workmgr-8548dc8745-zjzkp\" (UID: \"e0480e53-1328-4671-9168-ca55579c4e53\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.678048 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.678021    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mb42\" (UniqueName: \"kubernetes.io/projected/e0480e53-1328-4671-9168-ca55579c4e53-kube-api-access-6mb42\") pod \"klusterlet-addon-workmgr-8548dc8745-zjzkp\" (UID: \"e0480e53-1328-4671-9168-ca55579c4e53\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.704753 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.704724    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:49.824138 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:49.824106    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"]
Jul 09 04:01:49.826833 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:49.826801    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0480e53_1328_4671_9168_ca55579c4e53.slice/crio-3341a6c397d09fae7d97a670e2e9c3b3d6af010bdfeedafca45df6d418c661f5 WatchSource:0}: Error finding container 3341a6c397d09fae7d97a670e2e9c3b3d6af010bdfeedafca45df6d418c661f5: Status 404 returned error can't find the container with id 3341a6c397d09fae7d97a670e2e9c3b3d6af010bdfeedafca45df6d418c661f5
Jul 09 04:01:50.475518 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:50.475478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:50.476117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:50.476097    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1c2b0f3-4be4-4e34-975f-ced29f667c3d-service-ca-bundle\") pod \"router-default-5fd7cc5c49-nblmz\" (UID: \"f1c2b0f3-4be4-4e34-975f-ced29f667c3d\") " pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:50.633917 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:50.633874    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp" event={"ID":"e0480e53-1328-4671-9168-ca55579c4e53","Type":"ContainerStarted","Data":"3341a6c397d09fae7d97a670e2e9c3b3d6af010bdfeedafca45df6d418c661f5"}
Jul 09 04:01:50.726648 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:50.726498    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:50.860427 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:50.860391    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5fd7cc5c49-nblmz"]
Jul 09 04:01:50.864259 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:50.864226    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c2b0f3_4be4_4e34_975f_ced29f667c3d.slice/crio-b17605e24fe3d04034fa7a09b3728c6df8f88fc43f83c0bbb2b6a2b49d45ed63 WatchSource:0}: Error finding container b17605e24fe3d04034fa7a09b3728c6df8f88fc43f83c0bbb2b6a2b49d45ed63: Status 404 returned error can't find the container with id b17605e24fe3d04034fa7a09b3728c6df8f88fc43f83c0bbb2b6a2b49d45ed63
Jul 09 04:01:51.639717 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:51.639672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5fd7cc5c49-nblmz" event={"ID":"f1c2b0f3-4be4-4e34-975f-ced29f667c3d","Type":"ContainerStarted","Data":"22bd67efd66ce1596902a4a2aaa642e50310f6c93da00e5c3ca3f60c993012f0"}
Jul 09 04:01:51.640226 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:51.639769    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5fd7cc5c49-nblmz" event={"ID":"f1c2b0f3-4be4-4e34-975f-ced29f667c3d","Type":"ContainerStarted","Data":"b17605e24fe3d04034fa7a09b3728c6df8f88fc43f83c0bbb2b6a2b49d45ed63"}
Jul 09 04:01:51.663314 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:51.663258    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5fd7cc5c49-nblmz" podStartSLOduration=33.66324115 podStartE2EDuration="33.66324115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:51.661883868 +0000 UTC m=+118.036887705" watchObservedRunningTime="2026-07-09 04:01:51.66324115 +0000 UTC m=+118.038244964"
Jul 09 04:01:51.727224 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:51.727178    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:51.729848 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:51.729818    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:52.643227 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:52.643185    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:52.644408 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:52.644388    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5fd7cc5c49-nblmz"
Jul 09 04:01:53.798725 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:53.798676    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:53.800854 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:53.800832    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a511ac-2f5c-4fc2-904c-ffeb76048e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6z7dk\" (UID: \"17a511ac-2f5c-4fc2-904c-ffeb76048e4c\") " pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:53.896643 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:53.896605    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"]
Jul 09 04:01:53.898948 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:53.898933    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"
Jul 09 04:01:53.901578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:53.901559    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 09 04:01:53.901877 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:53.901862    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7nmj2\""
Jul 09 04:01:53.912216 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:53.912184    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"]
Jul 09 04:01:53.999609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:53.999578    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4c7adf19-91ea-4e32-8de8-a9f2ca6fcb7f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7rh4z\" (UID: \"4c7adf19-91ea-4e32-8de8-a9f2ca6fcb7f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"
Jul 09 04:01:54.061226 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.061128    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9vrhl\""
Jul 09 04:01:54.068925 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.068896    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6z7dk"
Jul 09 04:01:54.100335 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.100274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4c7adf19-91ea-4e32-8de8-a9f2ca6fcb7f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7rh4z\" (UID: \"4c7adf19-91ea-4e32-8de8-a9f2ca6fcb7f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"
Jul 09 04:01:54.103006 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.102981    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 09 04:01:54.113808 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.113779    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4c7adf19-91ea-4e32-8de8-a9f2ca6fcb7f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7rh4z\" (UID: \"4c7adf19-91ea-4e32-8de8-a9f2ca6fcb7f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"
Jul 09 04:01:54.208654 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.208609    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6z7dk"]
Jul 09 04:01:54.209822 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.209805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7nmj2\""
Jul 09 04:01:54.212099 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:54.212078    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a511ac_2f5c_4fc2_904c_ffeb76048e4c.slice/crio-e3b230ae8324c033fdbf965500e769b4a6f5d7650c80bdc987a35d13c9e601ff WatchSource:0}: Error finding container e3b230ae8324c033fdbf965500e769b4a6f5d7650c80bdc987a35d13c9e601ff: Status 404 returned error can't find the container with id e3b230ae8324c033fdbf965500e769b4a6f5d7650c80bdc987a35d13c9e601ff
Jul 09 04:01:54.217966 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.217942    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"
Jul 09 04:01:54.347704 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.347631    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"]
Jul 09 04:01:54.351428 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:01:54.351396    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7adf19_91ea_4e32_8de8_a9f2ca6fcb7f.slice/crio-d5b5138771aaab20cba148fd0657607c7f73a878d6bb503c83e9d7bf0bb4b97b WatchSource:0}: Error finding container d5b5138771aaab20cba148fd0657607c7f73a878d6bb503c83e9d7bf0bb4b97b: Status 404 returned error can't find the container with id d5b5138771aaab20cba148fd0657607c7f73a878d6bb503c83e9d7bf0bb4b97b
Jul 09 04:01:54.650180 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.650151    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z" event={"ID":"4c7adf19-91ea-4e32-8de8-a9f2ca6fcb7f","Type":"ContainerStarted","Data":"d5b5138771aaab20cba148fd0657607c7f73a878d6bb503c83e9d7bf0bb4b97b"}
Jul 09 04:01:54.651167 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:54.651145    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6z7dk" event={"ID":"17a511ac-2f5c-4fc2-904c-ffeb76048e4c","Type":"ContainerStarted","Data":"e3b230ae8324c033fdbf965500e769b4a6f5d7650c80bdc987a35d13c9e601ff"}
Jul 09 04:01:59.673836 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.673804    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp" event={"ID":"e0480e53-1328-4671-9168-ca55579c4e53","Type":"ContainerStarted","Data":"ee0859ded551ed19dc4b00129edd08dac84e39367e607f115948b518226321cf"}
Jul 09 04:01:59.674292 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.673991    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:59.675986 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.675952    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6z7dk" event={"ID":"17a511ac-2f5c-4fc2-904c-ffeb76048e4c","Type":"ContainerStarted","Data":"bb6559eea4a0ff45ef912f6e8c7e514f9187a440d08d0828a7c485b99ab0c389"}
Jul 09 04:01:59.676120 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.676040    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp"
Jul 09 04:01:59.677555 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.677518    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z" event={"ID":"4c7adf19-91ea-4e32-8de8-a9f2ca6fcb7f","Type":"ContainerStarted","Data":"c3e060753d92ea81841da4d519db40ee8650a3b188cdbc8fc4f8ed63a09ea9fe"}
Jul 09 04:01:59.677756 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.677713    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"
Jul 09 04:01:59.682621 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.682599    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z"
Jul 09 04:01:59.691598 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.691556    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8548dc8745-zjzkp" podStartSLOduration=1.785832324 podStartE2EDuration="10.691540143s" totalImagesPullingTime="8.905707819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:59.690784782 +0000 UTC m=+126.065788599" watchObservedRunningTime="2026-07-09 04:01:59.691540143 +0000 UTC m=+126.066543963"
Jul 09 04:01:59.721737 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.721690    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7rh4z" podStartSLOduration=2.340701294 podStartE2EDuration="6.721675726s" totalImagesPullingTime="4.380974432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:01:59.721534631 +0000 UTC m=+126.096538468" watchObservedRunningTime="2026-07-09 04:01:59.721675726 +0000 UTC m=+126.096679544"
Jul 09 04:01:59.948494 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.947114    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"]
Jul 09 04:01:59.952375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.950805    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:01:59.952375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.951959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2414171f-da52-4805-b5c0-65e2a3734654-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:01:59.952375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.951995    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27qq\" (UniqueName: \"kubernetes.io/projected/2414171f-da52-4805-b5c0-65e2a3734654-kube-api-access-c27qq\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:01:59.952375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.952055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2414171f-da52-4805-b5c0-65e2a3734654-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:01:59.952375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.952077    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2414171f-da52-4805-b5c0-65e2a3734654-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:01:59.959143 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.953440    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 09 04:01:59.959143 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.953704    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-skwwq\""
Jul 09 04:01:59.959143 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.954008    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 04:01:59.959143 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.956131    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 04:01:59.959143 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.956380    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 04:01:59.959143 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.954201    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 09 04:01:59.997975 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:01:59.997941    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"]
Jul 09 04:02:00.053312 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.053279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2414171f-da52-4805-b5c0-65e2a3734654-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.053501 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.053321    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2414171f-da52-4805-b5c0-65e2a3734654-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.053501 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.053358    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2414171f-da52-4805-b5c0-65e2a3734654-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.053501 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.053390    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27qq\" (UniqueName: \"kubernetes.io/projected/2414171f-da52-4805-b5c0-65e2a3734654-kube-api-access-c27qq\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.053501 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:02:00.053452    2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 09 04:02:00.053711 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:02:00.053521    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2414171f-da52-4805-b5c0-65e2a3734654-prometheus-operator-tls podName:2414171f-da52-4805-b5c0-65e2a3734654 nodeName:}" failed. No retries permitted until 2026-07-09 04:02:00.553502327 +0000 UTC m=+126.928506124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2414171f-da52-4805-b5c0-65e2a3734654-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-mh28d" (UID: "2414171f-da52-4805-b5c0-65e2a3734654") : secret "prometheus-operator-tls" not found
Jul 09 04:02:00.054105 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.054087    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2414171f-da52-4805-b5c0-65e2a3734654-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.055621 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.055592    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2414171f-da52-4805-b5c0-65e2a3734654-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.064047 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.064023    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27qq\" (UniqueName: \"kubernetes.io/projected/2414171f-da52-4805-b5c0-65e2a3734654-kube-api-access-c27qq\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.557359 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.557324    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2414171f-da52-4805-b5c0-65e2a3734654-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.560128 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.560096    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2414171f-da52-4805-b5c0-65e2a3734654-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mh28d\" (UID: \"2414171f-da52-4805-b5c0-65e2a3734654\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.567813 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.567789    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"
Jul 09 04:02:00.686173 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.686126    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6z7dk" event={"ID":"17a511ac-2f5c-4fc2-904c-ffeb76048e4c","Type":"ContainerStarted","Data":"fc61a64441011bd88c8e6840690aeb70d2c36af29274debcee90f2f470d75aa2"}
Jul 09 04:02:00.715185 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:00.715010    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d"]
Jul 09 04:02:00.718475 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:02:00.718442    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2414171f_da52_4805_b5c0_65e2a3734654.slice/crio-fde633421774e8a0e1b24be1d7de962bde57cb7f6be701f7ab4daf4f07ae4a10 WatchSource:0}: Error finding container fde633421774e8a0e1b24be1d7de962bde57cb7f6be701f7ab4daf4f07ae4a10: Status 404 returned error can't find the container with id fde633421774e8a0e1b24be1d7de962bde57cb7f6be701f7ab4daf4f07ae4a10
Jul 09 04:02:01.691285 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:01.691245    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d" event={"ID":"2414171f-da52-4805-b5c0-65e2a3734654","Type":"ContainerStarted","Data":"fde633421774e8a0e1b24be1d7de962bde57cb7f6be701f7ab4daf4f07ae4a10"}
Jul 09 04:02:03.700863 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:03.700826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d" event={"ID":"2414171f-da52-4805-b5c0-65e2a3734654","Type":"ContainerStarted","Data":"855a2bbaf61cadc006a25f4f8ffb8093c14c0763927a4aa19dc699d268d98ef9"}
Jul 09 04:02:03.701338 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:03.700930    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d" event={"ID":"2414171f-da52-4805-b5c0-65e2a3734654","Type":"ContainerStarted","Data":"620833a69305d2c2ed2f51536d91984815c1b99b8694f4f49accc91f1f0aff01"}
Jul 09 04:02:03.702856 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:03.702836    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6z7dk" event={"ID":"17a511ac-2f5c-4fc2-904c-ffeb76048e4c","Type":"ContainerStarted","Data":"f87560673ba045748337bef9d32bb5cedfecefa423e7d99da20959d4c0ed1900"}
Jul 09 04:02:03.722375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:03.722310    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mh28d" podStartSLOduration=2.766710611 podStartE2EDuration="4.722289348s" totalImagesPullingTime="1.955578737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:02:03.719760251 +0000 UTC m=+130.094764069" watchObservedRunningTime="2026-07-09 04:02:03.722289348 +0000 UTC m=+130.097293167"
Jul 09 04:02:03.743413 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:03.743363    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6z7dk" podStartSLOduration=34.392032234 podStartE2EDuration="42.743348186s" totalImagesPullingTime="8.351315952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:01:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:02:03.742489781 +0000 UTC m=+130.117493600" watchObservedRunningTime="2026-07-09 04:02:03.743348186 +0000 UTC m=+130.118352004"
Jul 09 04:02:03.884408 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:03.884369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:02:03.886724 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:03.886704    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca9faa-1fb7-4223-a246-f01696f726e4-metrics-certs\") pod \"network-metrics-daemon-h7zph\" (UID: \"29ca9faa-1fb7-4223-a246-f01696f726e4\") " pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:02:04.032259 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:04.032173    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fn82r\""
Jul 09 04:02:04.039567 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:04.039537    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7zph"
Jul 09 04:02:04.181086 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:04.181053    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7zph"]
Jul 09 04:02:04.183962 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:02:04.183929    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ca9faa_1fb7_4223_a246_f01696f726e4.slice/crio-88e56f7b551366817f67f766b0b9fe4cf9b88e97641f663de959a1eda82f9cf7 WatchSource:0}: Error finding container 88e56f7b551366817f67f766b0b9fe4cf9b88e97641f663de959a1eda82f9cf7: Status 404 returned error can't find the container with id 88e56f7b551366817f67f766b0b9fe4cf9b88e97641f663de959a1eda82f9cf7
Jul 09 04:02:04.708031 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:04.707996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7zph" event={"ID":"29ca9faa-1fb7-4223-a246-f01696f726e4","Type":"ContainerStarted","Data":"88e56f7b551366817f67f766b0b9fe4cf9b88e97641f663de959a1eda82f9cf7"}
Jul 09 04:02:05.712859 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:05.712825    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7zph" event={"ID":"29ca9faa-1fb7-4223-a246-f01696f726e4","Type":"ContainerStarted","Data":"75e28982c4a7473359b04b10d995b6a0b8a50d03c06eaad1231cfd1b8801c532"}
Jul 09 04:02:05.713273 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:05.712882    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7zph" event={"ID":"29ca9faa-1fb7-4223-a246-f01696f726e4","Type":"ContainerStarted","Data":"a89e3615418daded5979ec554ad11a89dd9dd712f36c89c8c4f73427fbe53a5a"}
Jul 09 04:02:05.730626 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:05.730573    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h7zph" podStartSLOduration=130.608895845 podStartE2EDuration="2m11.730558086s" totalImagesPullingTime="1.121662241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 03:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:02:05.728990887 +0000 UTC m=+132.103994718" watchObservedRunningTime="2026-07-09 04:02:05.730558086 +0000 UTC m=+132.105561904"
Jul 09 04:02:08.944171 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:08.944137    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wksm9"]
Jul 09 04:02:08.946921 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:08.946904    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:08.949593 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:08.949563    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 04:02:08.949717 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:08.949651    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 04:02:08.949717 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:08.949664    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g2w2m\""
Jul 09 04:02:08.949717 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:08.949686    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 04:02:09.020416 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.020374    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.020416 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.020420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00ee2827-a243-4364-bb8c-81d250338e94-sys\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.020641 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.020450    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9hh\" (UniqueName: \"kubernetes.io/projected/00ee2827-a243-4364-bb8c-81d250338e94-kube-api-access-9l9hh\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.020641 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.020486    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-accelerators-collector-config\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.020641 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.020543    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00ee2827-a243-4364-bb8c-81d250338e94-metrics-client-ca\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.020641 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.020575    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/00ee2827-a243-4364-bb8c-81d250338e94-root\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.020641 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.020602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-tls\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.020641 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.020622    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-wtmp\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.020641 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.020639    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-textfile\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121025 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.120997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00ee2827-a243-4364-bb8c-81d250338e94-sys\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121032    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9hh\" (UniqueName: \"kubernetes.io/projected/00ee2827-a243-4364-bb8c-81d250338e94-kube-api-access-9l9hh\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121059    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-accelerators-collector-config\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121078    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00ee2827-a243-4364-bb8c-81d250338e94-metrics-client-ca\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/00ee2827-a243-4364-bb8c-81d250338e94-root\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-tls\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121124    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00ee2827-a243-4364-bb8c-81d250338e94-sys\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121137    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-wtmp\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121188    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/00ee2827-a243-4364-bb8c-81d250338e94-root\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121190    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-textfile\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121696 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121696 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121406    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-wtmp\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121696 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:02:09.121508    2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 04:02:09.121696 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121578    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-textfile\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121696 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:02:09.121591    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-tls podName:00ee2827-a243-4364-bb8c-81d250338e94 nodeName:}" failed. No retries permitted until 2026-07-09 04:02:09.621569746 +0000 UTC m=+135.996573551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-tls") pod "node-exporter-wksm9" (UID: "00ee2827-a243-4364-bb8c-81d250338e94") : secret "node-exporter-tls" not found
Jul 09 04:02:09.121859 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121698    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00ee2827-a243-4364-bb8c-81d250338e94-metrics-client-ca\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.121859 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.121734    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-accelerators-collector-config\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.123522 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.123504    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.132095 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.132064    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9hh\" (UniqueName: \"kubernetes.io/projected/00ee2827-a243-4364-bb8c-81d250338e94-kube-api-access-9l9hh\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.624169 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.624131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-tls\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.626506 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.626483    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00ee2827-a243-4364-bb8c-81d250338e94-node-exporter-tls\") pod \"node-exporter-wksm9\" (UID: \"00ee2827-a243-4364-bb8c-81d250338e94\") " pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.855514 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:09.855477    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wksm9"
Jul 09 04:02:09.865372 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:02:09.865339    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ee2827_a243_4364_bb8c_81d250338e94.slice/crio-cfc1e654e6723db96a61e2926cd5a0a752e797672db686e978ed95e374010259 WatchSource:0}: Error finding container cfc1e654e6723db96a61e2926cd5a0a752e797672db686e978ed95e374010259: Status 404 returned error can't find the container with id cfc1e654e6723db96a61e2926cd5a0a752e797672db686e978ed95e374010259
Jul 09 04:02:10.730569 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.730533    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wksm9" event={"ID":"00ee2827-a243-4364-bb8c-81d250338e94","Type":"ContainerStarted","Data":"cfc1e654e6723db96a61e2926cd5a0a752e797672db686e978ed95e374010259"}
Jul 09 04:02:10.857335 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.857299    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-66985f7865-wbqlg"]
Jul 09 04:02:10.860140 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.860118    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:10.862786 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.862759    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 09 04:02:10.863189 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.863107    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 09 04:02:10.863383 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.863363    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-78cf7\""
Jul 09 04:02:10.863478 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.863459    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 09 04:02:10.863529 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.863472    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 09 04:02:10.863969 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.863944    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-70eo367s7hm1i\""
Jul 09 04:02:10.864172 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.864151    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 09 04:02:10.874391 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.874368    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66985f7865-wbqlg"]
Jul 09 04:02:10.933759 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.933721    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:10.933918 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.933774    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84kw\" (UniqueName: \"kubernetes.io/projected/3719dd95-45b2-4870-be21-4099d0f1b028-kube-api-access-w84kw\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:10.933918 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.933821    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:10.933918 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.933863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-grpc-tls\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:10.934086 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.933947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-tls\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:10.934086 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.934014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:10.934086 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.934047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3719dd95-45b2-4870-be21-4099d0f1b028-metrics-client-ca\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:10.934086 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:10.934071    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.034548 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.034456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.034548 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.034510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84kw\" (UniqueName: \"kubernetes.io/projected/3719dd95-45b2-4870-be21-4099d0f1b028-kube-api-access-w84kw\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.034548 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.034548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.034847 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.034572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-grpc-tls\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.034847 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.034601    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-tls\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.034847 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.034690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.034847 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.034736    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3719dd95-45b2-4870-be21-4099d0f1b028-metrics-client-ca\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.034847 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.034766    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.035660 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.035587    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3719dd95-45b2-4870-be21-4099d0f1b028-metrics-client-ca\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.037630 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.037600    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.038109 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.038066    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.038236 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.038142    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.038335 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.038315    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.038605 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.038570    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-thanos-querier-tls\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.039318 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.039296    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3719dd95-45b2-4870-be21-4099d0f1b028-secret-grpc-tls\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.043291 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.043271    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84kw\" (UniqueName: \"kubernetes.io/projected/3719dd95-45b2-4870-be21-4099d0f1b028-kube-api-access-w84kw\") pod \"thanos-querier-66985f7865-wbqlg\" (UID: \"3719dd95-45b2-4870-be21-4099d0f1b028\") " pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.171343 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.171319    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:11.305391 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.305314    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66985f7865-wbqlg"]
Jul 09 04:02:11.314037 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:02:11.314010    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3719dd95_45b2_4870_be21_4099d0f1b028.slice/crio-b367aa2685b20769a93826039d247a395b4cf24da47aa99ca173ef0b3f43b65f WatchSource:0}: Error finding container b367aa2685b20769a93826039d247a395b4cf24da47aa99ca173ef0b3f43b65f: Status 404 returned error can't find the container with id b367aa2685b20769a93826039d247a395b4cf24da47aa99ca173ef0b3f43b65f
Jul 09 04:02:11.734908 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.734866    2558 generic.go:356] "Generic (PLEG): container finished" podID="00ee2827-a243-4364-bb8c-81d250338e94" containerID="320b67c4b250f8e5943d2901496cef37d0e2d23aa20111d730ffd9b06af83363" exitCode=0
Jul 09 04:02:11.735397 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.734963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wksm9" event={"ID":"00ee2827-a243-4364-bb8c-81d250338e94","Type":"ContainerDied","Data":"320b67c4b250f8e5943d2901496cef37d0e2d23aa20111d730ffd9b06af83363"}
Jul 09 04:02:11.736767 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:11.736735    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg" event={"ID":"3719dd95-45b2-4870-be21-4099d0f1b028","Type":"ContainerStarted","Data":"b367aa2685b20769a93826039d247a395b4cf24da47aa99ca173ef0b3f43b65f"}
Jul 09 04:02:12.742601 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:12.742565    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wksm9" event={"ID":"00ee2827-a243-4364-bb8c-81d250338e94","Type":"ContainerStarted","Data":"0ab71867677b830fcc7fdc33414ca06539871d4d477a7e84a54cd81952974a36"}
Jul 09 04:02:12.743024 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:12.742616    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wksm9" event={"ID":"00ee2827-a243-4364-bb8c-81d250338e94","Type":"ContainerStarted","Data":"779900a60ac08283e423b8c6b75755ff68c43b00aa54bc4d6eec59a03adf7a51"}
Jul 09 04:02:12.765203 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:12.765150    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wksm9" podStartSLOduration=3.44760696 podStartE2EDuration="4.765134168s" totalImagesPullingTime="1.248008575s" totalInitContainerRuntime="69.518633ms" isStatefulPod=true podCreationTimestamp="2026-07-09 04:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:02:12.763606281 +0000 UTC m=+139.138610118" watchObservedRunningTime="2026-07-09 04:02:12.765134168 +0000 UTC m=+139.140137986"
Jul 09 04:02:13.179316 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.179277    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-66964b95c8-4xbjf"]
Jul 09 04:02:13.181824 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.181796    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.184493 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.184466    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dthtarn9ptrb1\""
Jul 09 04:02:13.184726 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.184706    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 04:02:13.185857 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.185838    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 09 04:02:13.186014 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.185920    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 09 04:02:13.186014 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.185941    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4mjpn\""
Jul 09 04:02:13.186014 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.185967    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 09 04:02:13.192902 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.192875    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66964b95c8-4xbjf"]
Jul 09 04:02:13.252867 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.252833    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/35bd7202-1875-4260-a2e7-00e0f2efc933-audit-log\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.252867 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.252867    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/35bd7202-1875-4260-a2e7-00e0f2efc933-client-ca-bundle\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.253073 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.252887    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9jw\" (UniqueName: \"kubernetes.io/projected/35bd7202-1875-4260-a2e7-00e0f2efc933-kube-api-access-4g9jw\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.253073 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.252911    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35bd7202-1875-4260-a2e7-00e0f2efc933-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.253073 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.253001    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/35bd7202-1875-4260-a2e7-00e0f2efc933-secret-metrics-server-tls\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.253073 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.253039    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/35bd7202-1875-4260-a2e7-00e0f2efc933-metrics-server-audit-profiles\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.253073 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.253065    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/35bd7202-1875-4260-a2e7-00e0f2efc933-secret-metrics-server-client-certs\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.354219 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.354178    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/35bd7202-1875-4260-a2e7-00e0f2efc933-audit-log\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.354375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.354252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/35bd7202-1875-4260-a2e7-00e0f2efc933-client-ca-bundle\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.354375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.354281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9jw\" (UniqueName: \"kubernetes.io/projected/35bd7202-1875-4260-a2e7-00e0f2efc933-kube-api-access-4g9jw\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.354375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.354309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35bd7202-1875-4260-a2e7-00e0f2efc933-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.354375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.354357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/35bd7202-1875-4260-a2e7-00e0f2efc933-secret-metrics-server-tls\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.354601 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.354388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/35bd7202-1875-4260-a2e7-00e0f2efc933-metrics-server-audit-profiles\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.354601 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.354420    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/35bd7202-1875-4260-a2e7-00e0f2efc933-secret-metrics-server-client-certs\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.354686 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.354669    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/35bd7202-1875-4260-a2e7-00e0f2efc933-audit-log\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.355307 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.355279    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35bd7202-1875-4260-a2e7-00e0f2efc933-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.355448 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.355415    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/35bd7202-1875-4260-a2e7-00e0f2efc933-metrics-server-audit-profiles\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.356857 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.356835    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/35bd7202-1875-4260-a2e7-00e0f2efc933-client-ca-bundle\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.356960 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.356871    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/35bd7202-1875-4260-a2e7-00e0f2efc933-secret-metrics-server-tls\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.356960 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.356915    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/35bd7202-1875-4260-a2e7-00e0f2efc933-secret-metrics-server-client-certs\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.362718 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.362694    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9jw\" (UniqueName: \"kubernetes.io/projected/35bd7202-1875-4260-a2e7-00e0f2efc933-kube-api-access-4g9jw\") pod \"metrics-server-66964b95c8-4xbjf\" (UID: \"35bd7202-1875-4260-a2e7-00e0f2efc933\") " pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:13.493623 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:13.493540    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:14.139295 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:14.139190    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66964b95c8-4xbjf"]
Jul 09 04:02:14.141995 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:02:14.141964    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bd7202_1875_4260_a2e7_00e0f2efc933.slice/crio-1577c2d9b0b848c1c4ca6d3689f48c15408c6f5f9acb7e410a1e3d2082af46c8 WatchSource:0}: Error finding container 1577c2d9b0b848c1c4ca6d3689f48c15408c6f5f9acb7e410a1e3d2082af46c8: Status 404 returned error can't find the container with id 1577c2d9b0b848c1c4ca6d3689f48c15408c6f5f9acb7e410a1e3d2082af46c8
Jul 09 04:02:14.753311 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:14.753237    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg" event={"ID":"3719dd95-45b2-4870-be21-4099d0f1b028","Type":"ContainerStarted","Data":"c9b3b1fe0c0209d561f1fca8f3a9e36baed9cce1ca319029f5bc75a426794461"}
Jul 09 04:02:14.753503 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:14.753360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg" event={"ID":"3719dd95-45b2-4870-be21-4099d0f1b028","Type":"ContainerStarted","Data":"2241b2e1fefdd877b5e969de818ec46e699be9cd85c64e19b19fc9d12188d60c"}
Jul 09 04:02:14.753503 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:14.753405    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg" event={"ID":"3719dd95-45b2-4870-be21-4099d0f1b028","Type":"ContainerStarted","Data":"ccdd5bc90d24d7765164d86b99d7c45acca315df8d5b9fb82a22cd8041e29789"}
Jul 09 04:02:14.754621 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:14.754595    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf" event={"ID":"35bd7202-1875-4260-a2e7-00e0f2efc933","Type":"ContainerStarted","Data":"1577c2d9b0b848c1c4ca6d3689f48c15408c6f5f9acb7e410a1e3d2082af46c8"}
Jul 09 04:02:15.762273 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:15.762240    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg" event={"ID":"3719dd95-45b2-4870-be21-4099d0f1b028","Type":"ContainerStarted","Data":"a7a2cd00e427b481e4449617deb728fa604bf34242eebd3ba72c8eeef608ee47"}
Jul 09 04:02:15.763669 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:15.763645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf" event={"ID":"35bd7202-1875-4260-a2e7-00e0f2efc933","Type":"ContainerStarted","Data":"99bc06585f0d2007318da0b5933c337716326661f7269a5b1e72e523133aa7cf"}
Jul 09 04:02:15.782127 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:15.782067    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf" podStartSLOduration=1.250874635 podStartE2EDuration="2.782045126s" totalImagesPullingTime="1.531170491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:02:15.780648995 +0000 UTC m=+142.155652832" watchObservedRunningTime="2026-07-09 04:02:15.782045126 +0000 UTC m=+142.157048946"
Jul 09 04:02:16.770473 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:16.770439    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg" event={"ID":"3719dd95-45b2-4870-be21-4099d0f1b028","Type":"ContainerStarted","Data":"33c1553680fb19f17f1439531d9123b0d565e8fb720e9d0793e140f19d0798ff"}
Jul 09 04:02:16.770828 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:16.770501    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg" event={"ID":"3719dd95-45b2-4870-be21-4099d0f1b028","Type":"ContainerStarted","Data":"3cb43576b676a7ee32601376239d9a257363d7fa5d852ce1c5ce7ca9a43d4142"}
Jul 09 04:02:16.795698 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:16.795646    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg" podStartSLOduration=2.616464446 podStartE2EDuration="6.795629761s" totalImagesPullingTime="4.179165315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:02:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:02:16.79360479 +0000 UTC m=+143.168608608" watchObservedRunningTime="2026-07-09 04:02:16.795629761 +0000 UTC m=+143.170633578"
Jul 09 04:02:17.774429 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:17.774397    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:21.736843 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.736806    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8d7d9f98c-j76k2"]
Jul 09 04:02:21.739711 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.739693    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.743448 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.743423    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 04:02:21.743570 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.743517    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 04:02:21.743570 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.743434    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 04:02:21.743672 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.743593    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2vtvr\""
Jul 09 04:02:21.743672 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.743644    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 04:02:21.743672 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.743662    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 04:02:21.743853 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.743832    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 04:02:21.743906 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.743856    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 04:02:21.751309 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.751288    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d7d9f98c-j76k2"]
Jul 09 04:02:21.824388 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.824332    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-oauth-serving-cert\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.824561 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.824398    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzsxb\" (UniqueName: \"kubernetes.io/projected/7715a079-ca96-4bda-8f59-58de19363ec3-kube-api-access-rzsxb\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.824561 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.824465    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-oauth-config\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.824561 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.824498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-service-ca\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.824561 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.824515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-console-config\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.824561 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.824535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-serving-cert\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.925389 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.925354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-oauth-serving-cert\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.925549 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.925409    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzsxb\" (UniqueName: \"kubernetes.io/projected/7715a079-ca96-4bda-8f59-58de19363ec3-kube-api-access-rzsxb\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.925549 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.925442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-oauth-config\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.925549 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.925469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-service-ca\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.925677 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.925649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-console-config\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.925726 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.925697    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-serving-cert\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.926150 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.926122    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-oauth-serving-cert\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.926301 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.926201    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-service-ca\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.926369 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.926320    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-console-config\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.927805 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.927774    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-oauth-config\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.927990 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.927972    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-serving-cert\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:21.934400 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:21.934379    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzsxb\" (UniqueName: \"kubernetes.io/projected/7715a079-ca96-4bda-8f59-58de19363ec3-kube-api-access-rzsxb\") pod \"console-8d7d9f98c-j76k2\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") " pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:22.049447 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:22.049350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:22.179519 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:22.179493    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d7d9f98c-j76k2"]
Jul 09 04:02:22.181915 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:02:22.181887    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7715a079_ca96_4bda_8f59_58de19363ec3.slice/crio-f249df944c529c9ff4b358530b2139d9bbc0dcd440132a2039c4292f5aed3063 WatchSource:0}: Error finding container f249df944c529c9ff4b358530b2139d9bbc0dcd440132a2039c4292f5aed3063: Status 404 returned error can't find the container with id f249df944c529c9ff4b358530b2139d9bbc0dcd440132a2039c4292f5aed3063
Jul 09 04:02:22.792152 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:22.792118    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d7d9f98c-j76k2" event={"ID":"7715a079-ca96-4bda-8f59-58de19363ec3","Type":"ContainerStarted","Data":"f249df944c529c9ff4b358530b2139d9bbc0dcd440132a2039c4292f5aed3063"}
Jul 09 04:02:23.785883 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:23.785853    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-66985f7865-wbqlg"
Jul 09 04:02:26.811906 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:26.811867    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d7d9f98c-j76k2" event={"ID":"7715a079-ca96-4bda-8f59-58de19363ec3","Type":"ContainerStarted","Data":"4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b"}
Jul 09 04:02:26.830747 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:26.830692    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8d7d9f98c-j76k2" podStartSLOduration=1.728885047 podStartE2EDuration="5.830677286s" totalImagesPullingTime="4.101792239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:02:26.830508802 +0000 UTC m=+153.205512620" watchObservedRunningTime="2026-07-09 04:02:26.830677286 +0000 UTC m=+153.205681103"
Jul 09 04:02:30.118678 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.118646    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c96dddfd9-b7rwx"]
Jul 09 04:02:30.241674 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.241643    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c96dddfd9-b7rwx"]
Jul 09 04:02:30.241839 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.241739    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.250061 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.250040    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 04:02:30.297665 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.297630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-oauth-config\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.297821 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.297699    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-trusted-ca-bundle\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.297821 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.297742    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-console-config\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.297821 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.297760    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85pp\" (UniqueName: \"kubernetes.io/projected/c983592f-cb0b-47f6-8585-3c6280991da9-kube-api-access-l85pp\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.297821 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.297787    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-oauth-serving-cert\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.297821 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.297815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-serving-cert\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.297992 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.297835    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-service-ca\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.398703 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.398670    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-trusted-ca-bundle\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.398837 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.398717    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-console-config\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.398837 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.398733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85pp\" (UniqueName: \"kubernetes.io/projected/c983592f-cb0b-47f6-8585-3c6280991da9-kube-api-access-l85pp\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.398837 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.398757    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-oauth-serving-cert\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.398980 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.398875    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-serving-cert\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.398980 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.398911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-service-ca\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.398980 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.398942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-oauth-config\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.399604 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.399573    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-oauth-serving-cert\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.399604 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.399600    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-console-config\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.399771 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.399585    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-service-ca\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.399811 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.399767    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-trusted-ca-bundle\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.401798 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.401774    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-oauth-config\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.401942 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.401920    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-serving-cert\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.406889 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.406867    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85pp\" (UniqueName: \"kubernetes.io/projected/c983592f-cb0b-47f6-8585-3c6280991da9-kube-api-access-l85pp\") pod \"console-5c96dddfd9-b7rwx\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") " pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.550582 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.550543    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:30.701759 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.701721    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c96dddfd9-b7rwx"]
Jul 09 04:02:30.704680 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:02:30.704658    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc983592f_cb0b_47f6_8585_3c6280991da9.slice/crio-ecbac642e7564cc3c218c780a8c0dbcd4d7a4572c7283a1da45e5186f677e25a WatchSource:0}: Error finding container ecbac642e7564cc3c218c780a8c0dbcd4d7a4572c7283a1da45e5186f677e25a: Status 404 returned error can't find the container with id ecbac642e7564cc3c218c780a8c0dbcd4d7a4572c7283a1da45e5186f677e25a
Jul 09 04:02:30.828314 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.828273    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c96dddfd9-b7rwx" event={"ID":"c983592f-cb0b-47f6-8585-3c6280991da9","Type":"ContainerStarted","Data":"1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4"}
Jul 09 04:02:30.828475 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.828375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c96dddfd9-b7rwx" event={"ID":"c983592f-cb0b-47f6-8585-3c6280991da9","Type":"ContainerStarted","Data":"ecbac642e7564cc3c218c780a8c0dbcd4d7a4572c7283a1da45e5186f677e25a"}
Jul 09 04:02:30.846966 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:30.846916    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c96dddfd9-b7rwx" podStartSLOduration=0.84690116 podStartE2EDuration="846.90116ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:02:30.846344595 +0000 UTC m=+157.221348414" watchObservedRunningTime="2026-07-09 04:02:30.84690116 +0000 UTC m=+157.221905030"
Jul 09 04:02:32.050369 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:32.050331    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:32.050756 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:32.050409    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:32.055291 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:32.055268    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:32.836901 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:32.836868    2558 generic.go:356] "Generic (PLEG): container finished" podID="66835b46-56de-45ff-9bf2-997eb83836f5" containerID="d1a527e97e89fdd55f9343723d7a9daaa1c3587d2ca8bc4ca0f5bf1c31f72286" exitCode=0
Jul 09 04:02:32.837068 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:32.836940    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq" event={"ID":"66835b46-56de-45ff-9bf2-997eb83836f5","Type":"ContainerDied","Data":"d1a527e97e89fdd55f9343723d7a9daaa1c3587d2ca8bc4ca0f5bf1c31f72286"}
Jul 09 04:02:32.837447 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:32.837428    2558 scope.go:117] "RemoveContainer" containerID="d1a527e97e89fdd55f9343723d7a9daaa1c3587d2ca8bc4ca0f5bf1c31f72286"
Jul 09 04:02:32.841190 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:32.841172    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:02:33.494268 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:33.494234    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:33.494763 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:33.494301    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:33.842558 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:33.842470    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-cfhxq" event={"ID":"66835b46-56de-45ff-9bf2-997eb83836f5","Type":"ContainerStarted","Data":"df8babbc7157d0b7be507f01d767bd0531fc4da18440bc12e3916e52f1be3c6f"}
Jul 09 04:02:40.551332 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:40.551289    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:40.551793 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:40.551359    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:40.555941 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:40.555919    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:40.872708 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:40.872631    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:02:40.918355 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:40.918319    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d7d9f98c-j76k2"]
Jul 09 04:02:53.499302 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:53.499221    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:02:53.503000 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:02:53.502975    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-66964b95c8-4xbjf"
Jul 09 04:03:05.945292 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:05.945233    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8d7d9f98c-j76k2" podUID="7715a079-ca96-4bda-8f59-58de19363ec3" containerName="console" containerID="cri-o://4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b" gracePeriod=15
Jul 09 04:03:06.187541 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.187510    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:03:06.299534 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.299438    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-console-config\") pod \"7715a079-ca96-4bda-8f59-58de19363ec3\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") "
Jul 09 04:03:06.299534 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.299501    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-oauth-config\") pod \"7715a079-ca96-4bda-8f59-58de19363ec3\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") "
Jul 09 04:03:06.299761 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.299541    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-serving-cert\") pod \"7715a079-ca96-4bda-8f59-58de19363ec3\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") "
Jul 09 04:03:06.299761 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.299571    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzsxb\" (UniqueName: \"kubernetes.io/projected/7715a079-ca96-4bda-8f59-58de19363ec3-kube-api-access-rzsxb\") pod \"7715a079-ca96-4bda-8f59-58de19363ec3\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") "
Jul 09 04:03:06.299761 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.299600    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-oauth-serving-cert\") pod \"7715a079-ca96-4bda-8f59-58de19363ec3\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") "
Jul 09 04:03:06.299761 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.299741    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-service-ca\") pod \"7715a079-ca96-4bda-8f59-58de19363ec3\" (UID: \"7715a079-ca96-4bda-8f59-58de19363ec3\") "
Jul 09 04:03:06.299964 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.299895    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-console-config" (OuterVolumeSpecName: "console-config") pod "7715a079-ca96-4bda-8f59-58de19363ec3" (UID: "7715a079-ca96-4bda-8f59-58de19363ec3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:03:06.300064 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.300034    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-console-config\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:03:06.300064 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.300047    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7715a079-ca96-4bda-8f59-58de19363ec3" (UID: "7715a079-ca96-4bda-8f59-58de19363ec3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:03:06.300245 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.300161    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-service-ca" (OuterVolumeSpecName: "service-ca") pod "7715a079-ca96-4bda-8f59-58de19363ec3" (UID: "7715a079-ca96-4bda-8f59-58de19363ec3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:03:06.301861 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.301841    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7715a079-ca96-4bda-8f59-58de19363ec3" (UID: "7715a079-ca96-4bda-8f59-58de19363ec3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:03:06.302155 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.302137    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7715a079-ca96-4bda-8f59-58de19363ec3" (UID: "7715a079-ca96-4bda-8f59-58de19363ec3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:03:06.302243 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.302145    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7715a079-ca96-4bda-8f59-58de19363ec3-kube-api-access-rzsxb" (OuterVolumeSpecName: "kube-api-access-rzsxb") pod "7715a079-ca96-4bda-8f59-58de19363ec3" (UID: "7715a079-ca96-4bda-8f59-58de19363ec3"). InnerVolumeSpecName "kube-api-access-rzsxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:03:06.400685 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.400641    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-oauth-config\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:03:06.400685 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.400681    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7715a079-ca96-4bda-8f59-58de19363ec3-console-serving-cert\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:03:06.400685 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.400691    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzsxb\" (UniqueName: \"kubernetes.io/projected/7715a079-ca96-4bda-8f59-58de19363ec3-kube-api-access-rzsxb\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:03:06.400685 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.400700    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-oauth-serving-cert\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:03:06.400941 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.400712    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7715a079-ca96-4bda-8f59-58de19363ec3-service-ca\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:03:06.973376 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.973344    2558 generic.go:356] "Generic (PLEG): container finished" podID="7715a079-ca96-4bda-8f59-58de19363ec3" containerID="4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b" exitCode=0
Jul 09 04:03:06.973848 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.973396    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d7d9f98c-j76k2"
Jul 09 04:03:06.973848 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.973433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d7d9f98c-j76k2" event={"ID":"7715a079-ca96-4bda-8f59-58de19363ec3","Type":"ContainerDied","Data":"4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b"}
Jul 09 04:03:06.973848 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.973487    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d7d9f98c-j76k2" event={"ID":"7715a079-ca96-4bda-8f59-58de19363ec3","Type":"ContainerDied","Data":"f249df944c529c9ff4b358530b2139d9bbc0dcd440132a2039c4292f5aed3063"}
Jul 09 04:03:06.973848 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.973510    2558 scope.go:117] "RemoveContainer" containerID="4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b"
Jul 09 04:03:06.981723 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.981703    2558 scope.go:117] "RemoveContainer" containerID="4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b"
Jul 09 04:03:06.981980 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:03:06.981962    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b\": container with ID starting with 4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b not found: ID does not exist" containerID="4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b"
Jul 09 04:03:06.982038 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.981988    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b"} err="failed to get container status \"4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b\": rpc error: code = NotFound desc = could not find container \"4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b\": container with ID starting with 4413e77c448feee8ddb222086ec438ca4f4c9097ed9fa59298628d1e13aa644b not found: ID does not exist"
Jul 09 04:03:06.993887 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.993858    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d7d9f98c-j76k2"]
Jul 09 04:03:06.997003 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:06.996981    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8d7d9f98c-j76k2"]
Jul 09 04:03:08.108412 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:08.108380    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7715a079-ca96-4bda-8f59-58de19363ec3" path="/var/lib/kubelet/pods/7715a079-ca96-4bda-8f59-58de19363ec3/volumes"
Jul 09 04:03:37.988992 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:37.988956    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5745cc6b96-szg6g"]
Jul 09 04:03:37.989574 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:37.989385    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7715a079-ca96-4bda-8f59-58de19363ec3" containerName="console"
Jul 09 04:03:37.989574 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:37.989404    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7715a079-ca96-4bda-8f59-58de19363ec3" containerName="console"
Jul 09 04:03:37.989574 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:37.989486    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7715a079-ca96-4bda-8f59-58de19363ec3" containerName="console"
Jul 09 04:03:37.992453 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:37.992434    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.004183 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.004158    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5745cc6b96-szg6g"]
Jul 09 04:03:38.156361 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.156327    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-oauth-config\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.156546 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.156373    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-console-config\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.156546 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.156402    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7wt\" (UniqueName: \"kubernetes.io/projected/9b4848c9-5705-4524-8000-dcb1851271bb-kube-api-access-wb7wt\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.156546 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.156456    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-trusted-ca-bundle\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.156546 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.156504    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-serving-cert\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.156546 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.156520    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-service-ca\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.156546 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.156544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-oauth-serving-cert\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.257052 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.256962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-console-config\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.257052 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257010    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7wt\" (UniqueName: \"kubernetes.io/projected/9b4848c9-5705-4524-8000-dcb1851271bb-kube-api-access-wb7wt\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.257052 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-trusted-ca-bundle\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.257350 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257073    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-serving-cert\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.257350 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-service-ca\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.257350 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257271    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-oauth-serving-cert\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.257509 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257420    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-oauth-config\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.257792 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257769    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-console-config\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.257932 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257909    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-oauth-serving-cert\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.258001 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257933    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-service-ca\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.258001 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.257951    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-trusted-ca-bundle\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.259670 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.259646    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-serving-cert\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.259752 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.259678    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-oauth-config\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.265219 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.265183    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7wt\" (UniqueName: \"kubernetes.io/projected/9b4848c9-5705-4524-8000-dcb1851271bb-kube-api-access-wb7wt\") pod \"console-5745cc6b96-szg6g\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") " pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.301717 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.301683    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:38.430885 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:38.430792    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5745cc6b96-szg6g"]
Jul 09 04:03:38.433414 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:03:38.433384    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4848c9_5705_4524_8000_dcb1851271bb.slice/crio-e5796ccf1422282e947bc05179cfadb4273a646669459ee9588eea19efa1dbf8 WatchSource:0}: Error finding container e5796ccf1422282e947bc05179cfadb4273a646669459ee9588eea19efa1dbf8: Status 404 returned error can't find the container with id e5796ccf1422282e947bc05179cfadb4273a646669459ee9588eea19efa1dbf8
Jul 09 04:03:39.092539 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:39.092499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5745cc6b96-szg6g" event={"ID":"9b4848c9-5705-4524-8000-dcb1851271bb","Type":"ContainerStarted","Data":"05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64"}
Jul 09 04:03:39.092919 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:39.092556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5745cc6b96-szg6g" event={"ID":"9b4848c9-5705-4524-8000-dcb1851271bb","Type":"ContainerStarted","Data":"e5796ccf1422282e947bc05179cfadb4273a646669459ee9588eea19efa1dbf8"}
Jul 09 04:03:39.110038 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:39.109982    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5745cc6b96-szg6g" podStartSLOduration=2.109969538 podStartE2EDuration="2.109969538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:03:39.109291145 +0000 UTC m=+225.484294995" watchObservedRunningTime="2026-07-09 04:03:39.109969538 +0000 UTC m=+225.484973356"
Jul 09 04:03:48.302863 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:48.302814    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:48.303360 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:48.302886    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:48.307641 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:48.307618    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:49.132130 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:49.132102    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:03:49.173961 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:03:49.173927    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c96dddfd9-b7rwx"]
Jul 09 04:04:14.197449 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.197400    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5c96dddfd9-b7rwx" podUID="c983592f-cb0b-47f6-8585-3c6280991da9" containerName="console" containerID="cri-o://1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4" gracePeriod=15
Jul 09 04:04:14.431293 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.431269    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:04:14.545358 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545279    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-oauth-serving-cert\") pod \"c983592f-cb0b-47f6-8585-3c6280991da9\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") "
Jul 09 04:04:14.545358 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545315    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-console-config\") pod \"c983592f-cb0b-47f6-8585-3c6280991da9\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") "
Jul 09 04:04:14.545358 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545349    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-oauth-config\") pod \"c983592f-cb0b-47f6-8585-3c6280991da9\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") "
Jul 09 04:04:14.545578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545472    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-serving-cert\") pod \"c983592f-cb0b-47f6-8585-3c6280991da9\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") "
Jul 09 04:04:14.545578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545517    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-trusted-ca-bundle\") pod \"c983592f-cb0b-47f6-8585-3c6280991da9\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") "
Jul 09 04:04:14.545578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545550    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l85pp\" (UniqueName: \"kubernetes.io/projected/c983592f-cb0b-47f6-8585-3c6280991da9-kube-api-access-l85pp\") pod \"c983592f-cb0b-47f6-8585-3c6280991da9\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") "
Jul 09 04:04:14.545578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545574    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-service-ca\") pod \"c983592f-cb0b-47f6-8585-3c6280991da9\" (UID: \"c983592f-cb0b-47f6-8585-3c6280991da9\") "
Jul 09 04:04:14.545769 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545707    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c983592f-cb0b-47f6-8585-3c6280991da9" (UID: "c983592f-cb0b-47f6-8585-3c6280991da9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:04:14.545769 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545743    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-console-config" (OuterVolumeSpecName: "console-config") pod "c983592f-cb0b-47f6-8585-3c6280991da9" (UID: "c983592f-cb0b-47f6-8585-3c6280991da9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:04:14.545933 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545907    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-oauth-serving-cert\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:04:14.546066 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545937    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-console-config\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:04:14.546066 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545974    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c983592f-cb0b-47f6-8585-3c6280991da9" (UID: "c983592f-cb0b-47f6-8585-3c6280991da9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:04:14.546066 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.545984    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-service-ca" (OuterVolumeSpecName: "service-ca") pod "c983592f-cb0b-47f6-8585-3c6280991da9" (UID: "c983592f-cb0b-47f6-8585-3c6280991da9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:04:14.547625 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.547596    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c983592f-cb0b-47f6-8585-3c6280991da9-kube-api-access-l85pp" (OuterVolumeSpecName: "kube-api-access-l85pp") pod "c983592f-cb0b-47f6-8585-3c6280991da9" (UID: "c983592f-cb0b-47f6-8585-3c6280991da9"). InnerVolumeSpecName "kube-api-access-l85pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:04:14.547901 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.547874    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c983592f-cb0b-47f6-8585-3c6280991da9" (UID: "c983592f-cb0b-47f6-8585-3c6280991da9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:04:14.547901 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.547889    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c983592f-cb0b-47f6-8585-3c6280991da9" (UID: "c983592f-cb0b-47f6-8585-3c6280991da9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:04:14.647072 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.647037    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-oauth-config\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:04:14.647072 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.647063    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c983592f-cb0b-47f6-8585-3c6280991da9-console-serving-cert\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:04:14.647072 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.647072    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-trusted-ca-bundle\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:04:14.647072 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.647081    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l85pp\" (UniqueName: \"kubernetes.io/projected/c983592f-cb0b-47f6-8585-3c6280991da9-kube-api-access-l85pp\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:04:14.647357 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:14.647090    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c983592f-cb0b-47f6-8585-3c6280991da9-service-ca\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:04:15.221103 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:15.221069    2558 generic.go:356] "Generic (PLEG): container finished" podID="c983592f-cb0b-47f6-8585-3c6280991da9" containerID="1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4" exitCode=0
Jul 09 04:04:15.221491 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:15.221113    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c96dddfd9-b7rwx"
Jul 09 04:04:15.221491 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:15.221157    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c96dddfd9-b7rwx" event={"ID":"c983592f-cb0b-47f6-8585-3c6280991da9","Type":"ContainerDied","Data":"1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4"}
Jul 09 04:04:15.221491 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:15.221221    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c96dddfd9-b7rwx" event={"ID":"c983592f-cb0b-47f6-8585-3c6280991da9","Type":"ContainerDied","Data":"ecbac642e7564cc3c218c780a8c0dbcd4d7a4572c7283a1da45e5186f677e25a"}
Jul 09 04:04:15.221491 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:15.221253    2558 scope.go:117] "RemoveContainer" containerID="1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4"
Jul 09 04:04:15.229343 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:15.229328    2558 scope.go:117] "RemoveContainer" containerID="1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4"
Jul 09 04:04:15.229602 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:04:15.229584    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4\": container with ID starting with 1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4 not found: ID does not exist" containerID="1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4"
Jul 09 04:04:15.229648 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:15.229612    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4"} err="failed to get container status \"1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4\": rpc error: code = NotFound desc = could not find container \"1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4\": container with ID starting with 1a4814285d1474fd7582a78ee5081d3107f9e0a3dccca7ed8ab5db0448a33de4 not found: ID does not exist"
Jul 09 04:04:15.243695 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:15.243668    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c96dddfd9-b7rwx"]
Jul 09 04:04:15.247505 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:15.247484    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c96dddfd9-b7rwx"]
Jul 09 04:04:16.108463 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:16.108431    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c983592f-cb0b-47f6-8585-3c6280991da9" path="/var/lib/kubelet/pods/c983592f-cb0b-47f6-8585-3c6280991da9/volumes"
Jul 09 04:04:54.023555 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:04:54.023525    2558 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 04:05:16.053172 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.053138    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c7888d595-p59g6"]
Jul 09 04:05:16.055463 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.053438    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c983592f-cb0b-47f6-8585-3c6280991da9" containerName="console"
Jul 09 04:05:16.055463 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.053450    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c983592f-cb0b-47f6-8585-3c6280991da9" containerName="console"
Jul 09 04:05:16.055463 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.053497    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c983592f-cb0b-47f6-8585-3c6280991da9" containerName="console"
Jul 09 04:05:16.056372 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.056357    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.069767 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.069746    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c7888d595-p59g6"]
Jul 09 04:05:16.129332 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.129296    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-oauth-config\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.129512 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.129340    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-console-config\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.129512 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.129368    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-serving-cert\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.129512 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.129476    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-oauth-serving-cert\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.129669 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.129523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntxtb\" (UniqueName: \"kubernetes.io/projected/45bdd57c-753f-4749-9d28-955b3d92a146-kube-api-access-ntxtb\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.129669 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.129571    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-service-ca\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.129669 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.129603    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-trusted-ca-bundle\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.229935 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.229897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxtb\" (UniqueName: \"kubernetes.io/projected/45bdd57c-753f-4749-9d28-955b3d92a146-kube-api-access-ntxtb\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.229935 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.229939    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-service-ca\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.230119 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.229962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-trusted-ca-bundle\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.230119 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.230002    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-oauth-config\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.230119 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.230047    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-console-config\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.230119 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.230086    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-serving-cert\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.230360 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.230131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-oauth-serving-cert\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.230798 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.230770    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-service-ca\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.230960 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.230921    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-oauth-serving-cert\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.230960 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.230861    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-console-config\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.231143 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.230967    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-trusted-ca-bundle\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.232550 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.232524    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-serving-cert\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.232659 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.232638    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-oauth-config\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.238553 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.238535    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxtb\" (UniqueName: \"kubernetes.io/projected/45bdd57c-753f-4749-9d28-955b3d92a146-kube-api-access-ntxtb\") pod \"console-7c7888d595-p59g6\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") " pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.365448 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.365362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:16.497366 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:16.497302    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c7888d595-p59g6"]
Jul 09 04:05:16.500086 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:05:16.500059    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bdd57c_753f_4749_9d28_955b3d92a146.slice/crio-2549c635461d4f6980c147f98d20cc48255a10864aca3c78a4a6930f96cf36d4 WatchSource:0}: Error finding container 2549c635461d4f6980c147f98d20cc48255a10864aca3c78a4a6930f96cf36d4: Status 404 returned error can't find the container with id 2549c635461d4f6980c147f98d20cc48255a10864aca3c78a4a6930f96cf36d4
Jul 09 04:05:17.465322 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:17.465283    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7888d595-p59g6" event={"ID":"45bdd57c-753f-4749-9d28-955b3d92a146","Type":"ContainerStarted","Data":"a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc"}
Jul 09 04:05:17.465679 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:17.465357    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7888d595-p59g6" event={"ID":"45bdd57c-753f-4749-9d28-955b3d92a146","Type":"ContainerStarted","Data":"2549c635461d4f6980c147f98d20cc48255a10864aca3c78a4a6930f96cf36d4"}
Jul 09 04:05:17.482898 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:17.482566    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c7888d595-p59g6" podStartSLOduration=1.482549109 podStartE2EDuration="1.482549109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:05:17.482022121 +0000 UTC m=+323.857025937" watchObservedRunningTime="2026-07-09 04:05:17.482549109 +0000 UTC m=+323.857552929"
Jul 09 04:05:26.366461 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:26.366423    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:26.366880 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:26.366485    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:26.371240 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:26.371194    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:26.501699 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:26.501671    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:05:26.551526 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:26.551483    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5745cc6b96-szg6g"]
Jul 09 04:05:51.573382 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.573274    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5745cc6b96-szg6g" podUID="9b4848c9-5705-4524-8000-dcb1851271bb" containerName="console" containerID="cri-o://05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64" gracePeriod=15
Jul 09 04:05:51.816457 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.816434    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:05:51.926166 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926126    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb7wt\" (UniqueName: \"kubernetes.io/projected/9b4848c9-5705-4524-8000-dcb1851271bb-kube-api-access-wb7wt\") pod \"9b4848c9-5705-4524-8000-dcb1851271bb\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") "
Jul 09 04:05:51.926370 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926183    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-trusted-ca-bundle\") pod \"9b4848c9-5705-4524-8000-dcb1851271bb\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") "
Jul 09 04:05:51.926370 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926234    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-oauth-config\") pod \"9b4848c9-5705-4524-8000-dcb1851271bb\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") "
Jul 09 04:05:51.926370 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926256    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-console-config\") pod \"9b4848c9-5705-4524-8000-dcb1851271bb\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") "
Jul 09 04:05:51.926370 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926289    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-service-ca\") pod \"9b4848c9-5705-4524-8000-dcb1851271bb\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") "
Jul 09 04:05:51.926370 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926345    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-serving-cert\") pod \"9b4848c9-5705-4524-8000-dcb1851271bb\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") "
Jul 09 04:05:51.926612 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926378    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-oauth-serving-cert\") pod \"9b4848c9-5705-4524-8000-dcb1851271bb\" (UID: \"9b4848c9-5705-4524-8000-dcb1851271bb\") "
Jul 09 04:05:51.926734 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926699    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b4848c9-5705-4524-8000-dcb1851271bb" (UID: "9b4848c9-5705-4524-8000-dcb1851271bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:05:51.926807 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926747    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-console-config" (OuterVolumeSpecName: "console-config") pod "9b4848c9-5705-4524-8000-dcb1851271bb" (UID: "9b4848c9-5705-4524-8000-dcb1851271bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:05:51.926807 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926760    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9b4848c9-5705-4524-8000-dcb1851271bb" (UID: "9b4848c9-5705-4524-8000-dcb1851271bb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:05:51.927054 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.926883    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b4848c9-5705-4524-8000-dcb1851271bb" (UID: "9b4848c9-5705-4524-8000-dcb1851271bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:05:51.928430 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.928409    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b4848c9-5705-4524-8000-dcb1851271bb" (UID: "9b4848c9-5705-4524-8000-dcb1851271bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:05:51.928830 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.928811    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4848c9-5705-4524-8000-dcb1851271bb-kube-api-access-wb7wt" (OuterVolumeSpecName: "kube-api-access-wb7wt") pod "9b4848c9-5705-4524-8000-dcb1851271bb" (UID: "9b4848c9-5705-4524-8000-dcb1851271bb"). InnerVolumeSpecName "kube-api-access-wb7wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:05:51.928830 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:51.928819    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b4848c9-5705-4524-8000-dcb1851271bb" (UID: "9b4848c9-5705-4524-8000-dcb1851271bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:05:52.027844 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.027801    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7wt\" (UniqueName: \"kubernetes.io/projected/9b4848c9-5705-4524-8000-dcb1851271bb-kube-api-access-wb7wt\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:05:52.027844 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.027832    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-trusted-ca-bundle\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:05:52.027844 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.027842    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-oauth-config\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:05:52.027844 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.027850    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-console-config\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:05:52.027844 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.027858    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-service-ca\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:05:52.028134 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.027865    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4848c9-5705-4524-8000-dcb1851271bb-console-serving-cert\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:05:52.028134 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.027873    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b4848c9-5705-4524-8000-dcb1851271bb-oauth-serving-cert\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:05:52.594390 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.594353    2558 generic.go:356] "Generic (PLEG): container finished" podID="9b4848c9-5705-4524-8000-dcb1851271bb" containerID="05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64" exitCode=0
Jul 09 04:05:52.594796 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.594396    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5745cc6b96-szg6g"
Jul 09 04:05:52.594796 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.594439    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5745cc6b96-szg6g" event={"ID":"9b4848c9-5705-4524-8000-dcb1851271bb","Type":"ContainerDied","Data":"05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64"}
Jul 09 04:05:52.594796 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.594488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5745cc6b96-szg6g" event={"ID":"9b4848c9-5705-4524-8000-dcb1851271bb","Type":"ContainerDied","Data":"e5796ccf1422282e947bc05179cfadb4273a646669459ee9588eea19efa1dbf8"}
Jul 09 04:05:52.594796 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.594507    2558 scope.go:117] "RemoveContainer" containerID="05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64"
Jul 09 04:05:52.602398 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.602374    2558 scope.go:117] "RemoveContainer" containerID="05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64"
Jul 09 04:05:52.602654 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:05:52.602637    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64\": container with ID starting with 05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64 not found: ID does not exist" containerID="05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64"
Jul 09 04:05:52.602706 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.602664    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64"} err="failed to get container status \"05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64\": rpc error: code = NotFound desc = could not find container \"05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64\": container with ID starting with 05b4b53b332e88c3527912e47db33a00aeb9465e02baa80f6973680f8e256c64 not found: ID does not exist"
Jul 09 04:05:52.609357 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.609338    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5745cc6b96-szg6g"]
Jul 09 04:05:52.612561 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:52.612542    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5745cc6b96-szg6g"]
Jul 09 04:05:54.109348 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:05:54.109312    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b4848c9-5705-4524-8000-dcb1851271bb" path="/var/lib/kubelet/pods/9b4848c9-5705-4524-8000-dcb1851271bb/volumes"
Jul 09 04:06:14.486253 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.486179    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"]
Jul 09 04:06:14.486710 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.486470    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4848c9-5705-4524-8000-dcb1851271bb" containerName="console"
Jul 09 04:06:14.486710 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.486482    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4848c9-5705-4524-8000-dcb1851271bb" containerName="console"
Jul 09 04:06:14.486710 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.486550    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4848c9-5705-4524-8000-dcb1851271bb" containerName="console"
Jul 09 04:06:14.490977 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.490961    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"
Jul 09 04:06:14.493467 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.493441    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 09 04:06:14.493609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.493521    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 09 04:06:14.493671 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.493624    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-j4fkr\""
Jul 09 04:06:14.503801 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.503779    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"]
Jul 09 04:06:14.507708 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.507683    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48ebe624-0651-4909-a6ad-16206dd819db-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjjvm\" (UID: \"48ebe624-0651-4909-a6ad-16206dd819db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"
Jul 09 04:06:14.507820 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.507715    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgbn6\" (UniqueName: \"kubernetes.io/projected/48ebe624-0651-4909-a6ad-16206dd819db-kube-api-access-wgbn6\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjjvm\" (UID: \"48ebe624-0651-4909-a6ad-16206dd819db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"
Jul 09 04:06:14.608943 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.608903    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48ebe624-0651-4909-a6ad-16206dd819db-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjjvm\" (UID: \"48ebe624-0651-4909-a6ad-16206dd819db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"
Jul 09 04:06:14.608943 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.608942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbn6\" (UniqueName: \"kubernetes.io/projected/48ebe624-0651-4909-a6ad-16206dd819db-kube-api-access-wgbn6\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjjvm\" (UID: \"48ebe624-0651-4909-a6ad-16206dd819db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"
Jul 09 04:06:14.609321 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.609300    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48ebe624-0651-4909-a6ad-16206dd819db-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjjvm\" (UID: \"48ebe624-0651-4909-a6ad-16206dd819db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"
Jul 09 04:06:14.617716 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.617692    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbn6\" (UniqueName: \"kubernetes.io/projected/48ebe624-0651-4909-a6ad-16206dd819db-kube-api-access-wgbn6\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjjvm\" (UID: \"48ebe624-0651-4909-a6ad-16206dd819db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"
Jul 09 04:06:14.799222 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.799131    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"
Jul 09 04:06:14.934806 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.934780    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm"]
Jul 09 04:06:14.937568 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:06:14.937543    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ebe624_0651_4909_a6ad_16206dd819db.slice/crio-7a12b85baf0516c0a3d49e05abc9a9358a5b6b2ab552c6f33796700f5a8e72d9 WatchSource:0}: Error finding container 7a12b85baf0516c0a3d49e05abc9a9358a5b6b2ab552c6f33796700f5a8e72d9: Status 404 returned error can't find the container with id 7a12b85baf0516c0a3d49e05abc9a9358a5b6b2ab552c6f33796700f5a8e72d9
Jul 09 04:06:14.940245 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:14.940228    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 04:06:15.677802 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:15.677763    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm" event={"ID":"48ebe624-0651-4909-a6ad-16206dd819db","Type":"ContainerStarted","Data":"7a12b85baf0516c0a3d49e05abc9a9358a5b6b2ab552c6f33796700f5a8e72d9"}
Jul 09 04:06:18.695388 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:18.695357    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm" event={"ID":"48ebe624-0651-4909-a6ad-16206dd819db","Type":"ContainerStarted","Data":"0b8cd904945d32eb0ad07bda30c14811e27f4a5e5f2a7b54cf8266255f5015d1"}
Jul 09 04:06:18.717670 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:18.717623    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjjvm" podStartSLOduration=1.759265601 podStartE2EDuration="4.717607009s" totalImagesPullingTime="2.958341408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:18.715266924 +0000 UTC m=+385.090270761" watchObservedRunningTime="2026-07-09 04:06:18.717607009 +0000 UTC m=+385.092610828"
Jul 09 04:06:26.036524 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.036486    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"]
Jul 09 04:06:26.039659 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.039643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"
Jul 09 04:06:26.042350 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.042328    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 09 04:06:26.042473 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.042407    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 09 04:06:26.043426 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.043411    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-82lnv\""
Jul 09 04:06:26.046300 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.046272    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"]
Jul 09 04:06:26.100353 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.100305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvq4n\" (UniqueName: \"kubernetes.io/projected/41f12818-8790-4433-bfd4-55db7c1680df-kube-api-access-nvq4n\") pod \"cert-manager-cainjector-559cdd8c87-rkjn2\" (UID: \"41f12818-8790-4433-bfd4-55db7c1680df\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"
Jul 09 04:06:26.100522 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.100363    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41f12818-8790-4433-bfd4-55db7c1680df-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rkjn2\" (UID: \"41f12818-8790-4433-bfd4-55db7c1680df\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"
Jul 09 04:06:26.201063 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.201025    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41f12818-8790-4433-bfd4-55db7c1680df-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rkjn2\" (UID: \"41f12818-8790-4433-bfd4-55db7c1680df\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"
Jul 09 04:06:26.201257 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.201183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvq4n\" (UniqueName: \"kubernetes.io/projected/41f12818-8790-4433-bfd4-55db7c1680df-kube-api-access-nvq4n\") pod \"cert-manager-cainjector-559cdd8c87-rkjn2\" (UID: \"41f12818-8790-4433-bfd4-55db7c1680df\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"
Jul 09 04:06:26.211827 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.211799    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41f12818-8790-4433-bfd4-55db7c1680df-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rkjn2\" (UID: \"41f12818-8790-4433-bfd4-55db7c1680df\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"
Jul 09 04:06:26.212075 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.212053    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvq4n\" (UniqueName: \"kubernetes.io/projected/41f12818-8790-4433-bfd4-55db7c1680df-kube-api-access-nvq4n\") pod \"cert-manager-cainjector-559cdd8c87-rkjn2\" (UID: \"41f12818-8790-4433-bfd4-55db7c1680df\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"
Jul 09 04:06:26.358382 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.358278    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"
Jul 09 04:06:26.487679 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.487649    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2"]
Jul 09 04:06:26.490625 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:06:26.490600    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41f12818_8790_4433_bfd4_55db7c1680df.slice/crio-0fc8c03a6a90d93abb60f3de79c44c964a72b8db424034974d93269ee3910473 WatchSource:0}: Error finding container 0fc8c03a6a90d93abb60f3de79c44c964a72b8db424034974d93269ee3910473: Status 404 returned error can't find the container with id 0fc8c03a6a90d93abb60f3de79c44c964a72b8db424034974d93269ee3910473
Jul 09 04:06:26.728653 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:26.728615    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2" event={"ID":"41f12818-8790-4433-bfd4-55db7c1680df","Type":"ContainerStarted","Data":"0fc8c03a6a90d93abb60f3de79c44c964a72b8db424034974d93269ee3910473"}
Jul 09 04:06:30.746519 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:30.746481    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2" event={"ID":"41f12818-8790-4433-bfd4-55db7c1680df","Type":"ContainerStarted","Data":"3d8bfc0ff91ad463d602abf8e3cce00881ceed8c5a987afb3536f068dde34420"}
Jul 09 04:06:30.778609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:30.778561    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rkjn2" podStartSLOduration=1.204199736 podStartE2EDuration="4.778521437s" totalImagesPullingTime="3.574321701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:30.778387322 +0000 UTC m=+397.153391141" watchObservedRunningTime="2026-07-09 04:06:30.778521437 +0000 UTC m=+397.153525255"
Jul 09 04:06:32.254304 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.254269    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"]
Jul 09 04:06:32.257524 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.257509    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"
Jul 09 04:06:32.260184 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.260162    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9x7hm\""
Jul 09 04:06:32.260435 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.260419    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 04:06:32.261180 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.261163    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 04:06:32.269048 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.269031    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"]
Jul 09 04:06:32.353637 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.353602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxpcv\" (UniqueName: \"kubernetes.io/projected/9c3fbdb0-639e-44a5-88e3-043bf26e3471-kube-api-access-vxpcv\") pod \"openshift-lws-operator-fd8ccff4c-p79j8\" (UID: \"9c3fbdb0-639e-44a5-88e3-043bf26e3471\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"
Jul 09 04:06:32.353823 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.353645    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3fbdb0-639e-44a5-88e3-043bf26e3471-tmp\") pod \"openshift-lws-operator-fd8ccff4c-p79j8\" (UID: \"9c3fbdb0-639e-44a5-88e3-043bf26e3471\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"
Jul 09 04:06:32.455085 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.455052    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxpcv\" (UniqueName: \"kubernetes.io/projected/9c3fbdb0-639e-44a5-88e3-043bf26e3471-kube-api-access-vxpcv\") pod \"openshift-lws-operator-fd8ccff4c-p79j8\" (UID: \"9c3fbdb0-639e-44a5-88e3-043bf26e3471\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"
Jul 09 04:06:32.455314 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.455095    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3fbdb0-639e-44a5-88e3-043bf26e3471-tmp\") pod \"openshift-lws-operator-fd8ccff4c-p79j8\" (UID: \"9c3fbdb0-639e-44a5-88e3-043bf26e3471\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"
Jul 09 04:06:32.455483 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.455464    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3fbdb0-639e-44a5-88e3-043bf26e3471-tmp\") pod \"openshift-lws-operator-fd8ccff4c-p79j8\" (UID: \"9c3fbdb0-639e-44a5-88e3-043bf26e3471\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"
Jul 09 04:06:32.463879 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.463847    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxpcv\" (UniqueName: \"kubernetes.io/projected/9c3fbdb0-639e-44a5-88e3-043bf26e3471-kube-api-access-vxpcv\") pod \"openshift-lws-operator-fd8ccff4c-p79j8\" (UID: \"9c3fbdb0-639e-44a5-88e3-043bf26e3471\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"
Jul 09 04:06:32.565974 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.565882    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"
Jul 09 04:06:32.695056 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.695024    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8"]
Jul 09 04:06:32.697741 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:06:32.697710    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c3fbdb0_639e_44a5_88e3_043bf26e3471.slice/crio-581971bee38b6bb0094845593062925e5bec733d3f202246fcbc9910cec40a6e WatchSource:0}: Error finding container 581971bee38b6bb0094845593062925e5bec733d3f202246fcbc9910cec40a6e: Status 404 returned error can't find the container with id 581971bee38b6bb0094845593062925e5bec733d3f202246fcbc9910cec40a6e
Jul 09 04:06:32.755473 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:32.755443    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8" event={"ID":"9c3fbdb0-639e-44a5-88e3-043bf26e3471","Type":"ContainerStarted","Data":"581971bee38b6bb0094845593062925e5bec733d3f202246fcbc9910cec40a6e"}
Jul 09 04:06:36.780115 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:36.780070    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8" event={"ID":"9c3fbdb0-639e-44a5-88e3-043bf26e3471","Type":"ContainerStarted","Data":"4c67c95a7286f137e3e30c5b2882be8191adf8e9434e0d4a5be16561ac77a9b0"}
Jul 09 04:06:36.797057 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:36.797010    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-p79j8" podStartSLOduration=1.54916845 podStartE2EDuration="4.796996618s" totalImagesPullingTime="3.247828168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:06:36.796292409 +0000 UTC m=+403.171296229" watchObservedRunningTime="2026-07-09 04:06:36.796996618 +0000 UTC m=+403.172000435"
Jul 09 04:06:56.189671 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.189636    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"]
Jul 09 04:06:56.196436 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.196410    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.199869 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.199845    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 09 04:06:56.200007 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.199900    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 09 04:06:56.200007 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.199928    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 09 04:06:56.200136 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.200122    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7rq67\""
Jul 09 04:06:56.201084 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.201068    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 09 04:06:56.215387 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.215361    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"]
Jul 09 04:06:56.348521 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.348484    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/888128d3-c724-47a8-b1fd-a0277e8c028c-webhook-cert\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.348700 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.348542    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/888128d3-c724-47a8-b1fd-a0277e8c028c-manifests\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.348700 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.348562    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86kxd\" (UniqueName: \"kubernetes.io/projected/888128d3-c724-47a8-b1fd-a0277e8c028c-kube-api-access-86kxd\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.348700 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.348646    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888128d3-c724-47a8-b1fd-a0277e8c028c-tmp\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.348700 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.348674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/888128d3-c724-47a8-b1fd-a0277e8c028c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.449889 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.449810    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/888128d3-c724-47a8-b1fd-a0277e8c028c-manifests\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.449889 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.449844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86kxd\" (UniqueName: \"kubernetes.io/projected/888128d3-c724-47a8-b1fd-a0277e8c028c-kube-api-access-86kxd\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.450117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.449894    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888128d3-c724-47a8-b1fd-a0277e8c028c-tmp\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.450117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.449913    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/888128d3-c724-47a8-b1fd-a0277e8c028c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.450117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.449934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/888128d3-c724-47a8-b1fd-a0277e8c028c-webhook-cert\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.450318 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.450193    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/888128d3-c724-47a8-b1fd-a0277e8c028c-manifests\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.452324 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.452295    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888128d3-c724-47a8-b1fd-a0277e8c028c-tmp\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.452461 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.452358    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/888128d3-c724-47a8-b1fd-a0277e8c028c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.452461 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.452434    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/888128d3-c724-47a8-b1fd-a0277e8c028c-webhook-cert\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.459984 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.459965    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86kxd\" (UniqueName: \"kubernetes.io/projected/888128d3-c724-47a8-b1fd-a0277e8c028c-kube-api-access-86kxd\") pod \"opendatahub-operator-controller-manager-df844f965-59c7h\" (UID: \"888128d3-c724-47a8-b1fd-a0277e8c028c\") " pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.505971 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.505939    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:06:56.662202 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.662177    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"]
Jul 09 04:06:56.664100 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:06:56.664074    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888128d3_c724_47a8_b1fd_a0277e8c028c.slice/crio-3d58d94a4156cf7210d20e9e7b496c0de0709b7538f0f630d7210c0e44b845f6 WatchSource:0}: Error finding container 3d58d94a4156cf7210d20e9e7b496c0de0709b7538f0f630d7210c0e44b845f6: Status 404 returned error can't find the container with id 3d58d94a4156cf7210d20e9e7b496c0de0709b7538f0f630d7210c0e44b845f6
Jul 09 04:06:56.862424 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:56.862336    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h" event={"ID":"888128d3-c724-47a8-b1fd-a0277e8c028c","Type":"ContainerStarted","Data":"3d58d94a4156cf7210d20e9e7b496c0de0709b7538f0f630d7210c0e44b845f6"}
Jul 09 04:06:59.882749 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:06:59.882718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h" event={"ID":"888128d3-c724-47a8-b1fd-a0277e8c028c","Type":"ContainerStarted","Data":"69c2c6eea9b7d4125b805bd1f2184c3af6b1a2d72610a3cc1eb02aee77429c5d"}
Jul 09 04:07:00.887793 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:00.887757    2558 generic.go:356] "Generic (PLEG): container finished" podID="888128d3-c724-47a8-b1fd-a0277e8c028c" containerID="69c2c6eea9b7d4125b805bd1f2184c3af6b1a2d72610a3cc1eb02aee77429c5d" exitCode=0
Jul 09 04:07:00.888143 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:00.887840    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h" event={"ID":"888128d3-c724-47a8-b1fd-a0277e8c028c","Type":"ContainerDied","Data":"69c2c6eea9b7d4125b805bd1f2184c3af6b1a2d72610a3cc1eb02aee77429c5d"}
Jul 09 04:07:01.895852 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:01.895815    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h" event={"ID":"888128d3-c724-47a8-b1fd-a0277e8c028c","Type":"ContainerStarted","Data":"444b3f843a0b4c43c8dcda2c538e58e5518b24081b1a15f6c54209809825d811"}
Jul 09 04:07:01.896283 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:01.895969    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:07:01.919536 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:01.919493    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h" podStartSLOduration=1.374735638 podStartE2EDuration="5.919477121s" totalImagesPullingTime="3.535906936s" totalInitContainerRuntime="1.008834547s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:06:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:01.918374654 +0000 UTC m=+428.293378472" watchObservedRunningTime="2026-07-09 04:07:01.919477121 +0000 UTC m=+428.294480939"
Jul 09 04:07:12.902039 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:12.901994    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-df844f965-59c7h"
Jul 09 04:07:16.092238 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.092186    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2l5z7"]
Jul 09 04:07:16.099930 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.099902    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:16.102763 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.102531    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7c7zf\""
Jul 09 04:07:16.103685 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.103641    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 09 04:07:16.113264 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.113241    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2l5z7"]
Jul 09 04:07:16.211640 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.211554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55nxw\" (UniqueName: \"kubernetes.io/projected/5fd4763c-f863-46ec-a3d9-2b75a89890ff-kube-api-access-55nxw\") pod \"odh-model-controller-7998c67dc4-2l5z7\" (UID: \"5fd4763c-f863-46ec-a3d9-2b75a89890ff\") " pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:16.211774 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.211642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd4763c-f863-46ec-a3d9-2b75a89890ff-cert\") pod \"odh-model-controller-7998c67dc4-2l5z7\" (UID: \"5fd4763c-f863-46ec-a3d9-2b75a89890ff\") " pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:16.312664 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.312622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd4763c-f863-46ec-a3d9-2b75a89890ff-cert\") pod \"odh-model-controller-7998c67dc4-2l5z7\" (UID: \"5fd4763c-f863-46ec-a3d9-2b75a89890ff\") " pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:16.312838 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.312722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55nxw\" (UniqueName: \"kubernetes.io/projected/5fd4763c-f863-46ec-a3d9-2b75a89890ff-kube-api-access-55nxw\") pod \"odh-model-controller-7998c67dc4-2l5z7\" (UID: \"5fd4763c-f863-46ec-a3d9-2b75a89890ff\") " pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:16.315187 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.315165    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd4763c-f863-46ec-a3d9-2b75a89890ff-cert\") pod \"odh-model-controller-7998c67dc4-2l5z7\" (UID: \"5fd4763c-f863-46ec-a3d9-2b75a89890ff\") " pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:16.321056 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.321029    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nxw\" (UniqueName: \"kubernetes.io/projected/5fd4763c-f863-46ec-a3d9-2b75a89890ff-kube-api-access-55nxw\") pod \"odh-model-controller-7998c67dc4-2l5z7\" (UID: \"5fd4763c-f863-46ec-a3d9-2b75a89890ff\") " pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:16.413194 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.413157    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:16.544714 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.544682    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2l5z7"]
Jul 09 04:07:16.547949 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:07:16.547924    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd4763c_f863_46ec_a3d9_2b75a89890ff.slice/crio-eda45c65af850bec200a3a6f5efd99d98cfda6cd9a0fe1f05c9294a10ae2cfa9 WatchSource:0}: Error finding container eda45c65af850bec200a3a6f5efd99d98cfda6cd9a0fe1f05c9294a10ae2cfa9: Status 404 returned error can't find the container with id eda45c65af850bec200a3a6f5efd99d98cfda6cd9a0fe1f05c9294a10ae2cfa9
Jul 09 04:07:16.959882 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:16.959810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7" event={"ID":"5fd4763c-f863-46ec-a3d9-2b75a89890ff","Type":"ContainerStarted","Data":"eda45c65af850bec200a3a6f5efd99d98cfda6cd9a0fe1f05c9294a10ae2cfa9"}
Jul 09 04:07:18.525695 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.525654    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"]
Jul 09 04:07:18.533848 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.533810    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.536711 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.536684    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-87rxd\""
Jul 09 04:07:18.536949 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.536930    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 09 04:07:18.537104 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.537087    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 09 04:07:18.537449 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.537423    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"]
Jul 09 04:07:18.629447 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.629398    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxb2j\" (UniqueName: \"kubernetes.io/projected/09b427f1-f906-43a0-83fe-99fb1646ef7b-kube-api-access-sxb2j\") pod \"kube-auth-proxy-779b7dc9c4-shdwq\" (UID: \"09b427f1-f906-43a0-83fe-99fb1646ef7b\") " pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.629631 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.629495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09b427f1-f906-43a0-83fe-99fb1646ef7b-tmp\") pod \"kube-auth-proxy-779b7dc9c4-shdwq\" (UID: \"09b427f1-f906-43a0-83fe-99fb1646ef7b\") " pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.629631 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.629543    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b427f1-f906-43a0-83fe-99fb1646ef7b-tls-certs\") pod \"kube-auth-proxy-779b7dc9c4-shdwq\" (UID: \"09b427f1-f906-43a0-83fe-99fb1646ef7b\") " pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.730121 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.730059    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09b427f1-f906-43a0-83fe-99fb1646ef7b-tmp\") pod \"kube-auth-proxy-779b7dc9c4-shdwq\" (UID: \"09b427f1-f906-43a0-83fe-99fb1646ef7b\") " pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.730336 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.730130    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b427f1-f906-43a0-83fe-99fb1646ef7b-tls-certs\") pod \"kube-auth-proxy-779b7dc9c4-shdwq\" (UID: \"09b427f1-f906-43a0-83fe-99fb1646ef7b\") " pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.730336 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.730239    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxb2j\" (UniqueName: \"kubernetes.io/projected/09b427f1-f906-43a0-83fe-99fb1646ef7b-kube-api-access-sxb2j\") pod \"kube-auth-proxy-779b7dc9c4-shdwq\" (UID: \"09b427f1-f906-43a0-83fe-99fb1646ef7b\") " pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.732941 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.732917    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09b427f1-f906-43a0-83fe-99fb1646ef7b-tmp\") pod \"kube-auth-proxy-779b7dc9c4-shdwq\" (UID: \"09b427f1-f906-43a0-83fe-99fb1646ef7b\") " pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.733104 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.733075    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b427f1-f906-43a0-83fe-99fb1646ef7b-tls-certs\") pod \"kube-auth-proxy-779b7dc9c4-shdwq\" (UID: \"09b427f1-f906-43a0-83fe-99fb1646ef7b\") " pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.739564 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.739539    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxb2j\" (UniqueName: \"kubernetes.io/projected/09b427f1-f906-43a0-83fe-99fb1646ef7b-kube-api-access-sxb2j\") pod \"kube-auth-proxy-779b7dc9c4-shdwq\" (UID: \"09b427f1-f906-43a0-83fe-99fb1646ef7b\") " pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:18.846225 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:18.846127    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"
Jul 09 04:07:19.641325 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:19.641165    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq"]
Jul 09 04:07:19.644387 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:07:19.644359    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b427f1_f906_43a0_83fe_99fb1646ef7b.slice/crio-070d5927203a2a554f003bd27cd5c3f10b4635243b26c42754262642690cdabe WatchSource:0}: Error finding container 070d5927203a2a554f003bd27cd5c3f10b4635243b26c42754262642690cdabe: Status 404 returned error can't find the container with id 070d5927203a2a554f003bd27cd5c3f10b4635243b26c42754262642690cdabe
Jul 09 04:07:19.979960 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:19.979924    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq" event={"ID":"09b427f1-f906-43a0-83fe-99fb1646ef7b","Type":"ContainerStarted","Data":"070d5927203a2a554f003bd27cd5c3f10b4635243b26c42754262642690cdabe"}
Jul 09 04:07:19.981469 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:19.981430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7" event={"ID":"5fd4763c-f863-46ec-a3d9-2b75a89890ff","Type":"ContainerStarted","Data":"01687e5d42fde0845f51f0e0e0916d27b9eeadd2e9d7dc702330dd3e067e8a3f"}
Jul 09 04:07:19.981594 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:19.981572    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:19.998414 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:19.998367    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7" podStartSLOduration=1.003508396 podStartE2EDuration="3.998352985s" totalImagesPullingTime="2.994844589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:19.996163915 +0000 UTC m=+446.371167737" watchObservedRunningTime="2026-07-09 04:07:19.998352985 +0000 UTC m=+446.373356835"
Jul 09 04:07:22.998631 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:22.998583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq" event={"ID":"09b427f1-f906-43a0-83fe-99fb1646ef7b","Type":"ContainerStarted","Data":"8491d3c471ff3fe4658aef7b7f643802309b7fa5540c12deb72a48b19a256774"}
Jul 09 04:07:23.014408 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.014349    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-779b7dc9c4-shdwq" podStartSLOduration=2.457432955 podStartE2EDuration="5.014328799s" totalImagesPullingTime="2.556895844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:23.014093803 +0000 UTC m=+449.389097640" watchObservedRunningTime="2026-07-09 04:07:23.014328799 +0000 UTC m=+449.389332618"
Jul 09 04:07:23.604661 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.604623    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"]
Jul 09 04:07:23.607885 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.607861    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.611688 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.611666    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 04:07:23.611843 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.611820    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ct2c6\""
Jul 09 04:07:23.611931 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.611833    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 04:07:23.611931 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.611834    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 04:07:23.619405 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.619382    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"]
Jul 09 04:07:23.672550 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.672507    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b158fc3-0386-47b2-8159-f583fbd7a1b9-cert\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.672550 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.672556    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b158fc3-0386-47b2-8159-f583fbd7a1b9-metrics-cert\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.672786 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.672618    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b158fc3-0386-47b2-8159-f583fbd7a1b9-manager-config\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.672786 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.672641    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s47sj\" (UniqueName: \"kubernetes.io/projected/5b158fc3-0386-47b2-8159-f583fbd7a1b9-kube-api-access-s47sj\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.773837 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.773802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b158fc3-0386-47b2-8159-f583fbd7a1b9-manager-config\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.773837 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.773845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s47sj\" (UniqueName: \"kubernetes.io/projected/5b158fc3-0386-47b2-8159-f583fbd7a1b9-kube-api-access-s47sj\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.774053 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.773880    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b158fc3-0386-47b2-8159-f583fbd7a1b9-cert\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.774053 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.773899    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b158fc3-0386-47b2-8159-f583fbd7a1b9-metrics-cert\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.774528 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.774504    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b158fc3-0386-47b2-8159-f583fbd7a1b9-manager-config\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.776354 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.776329    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b158fc3-0386-47b2-8159-f583fbd7a1b9-metrics-cert\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.776448 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.776374    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b158fc3-0386-47b2-8159-f583fbd7a1b9-cert\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.792579 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.792554    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s47sj\" (UniqueName: \"kubernetes.io/projected/5b158fc3-0386-47b2-8159-f583fbd7a1b9-kube-api-access-s47sj\") pod \"lws-controller-manager-765cd6b57c-lcvtc\" (UID: \"5b158fc3-0386-47b2-8159-f583fbd7a1b9\") " pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:23.917749 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:23.917703    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:24.072601 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:24.072555    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"]
Jul 09 04:07:24.075572 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:07:24.075543    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b158fc3_0386_47b2_8159_f583fbd7a1b9.slice/crio-54a7773b4ff55aea9c39e0fc4570165ae2d5414c5995d2081a8171d6a7c36159 WatchSource:0}: Error finding container 54a7773b4ff55aea9c39e0fc4570165ae2d5414c5995d2081a8171d6a7c36159: Status 404 returned error can't find the container with id 54a7773b4ff55aea9c39e0fc4570165ae2d5414c5995d2081a8171d6a7c36159
Jul 09 04:07:25.010717 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:25.010681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc" event={"ID":"5b158fc3-0386-47b2-8159-f583fbd7a1b9","Type":"ContainerStarted","Data":"54a7773b4ff55aea9c39e0fc4570165ae2d5414c5995d2081a8171d6a7c36159"}
Jul 09 04:07:26.016857 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:26.016825    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc" event={"ID":"5b158fc3-0386-47b2-8159-f583fbd7a1b9","Type":"ContainerStarted","Data":"10348a3e1e3766dba26e957b8c4cb4659cdd5f05e2c44ce1f9186fffa7d9b27e"}
Jul 09 04:07:26.017284 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:26.016936    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:07:26.040528 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:26.040476    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc" podStartSLOduration=1.557365766 podStartE2EDuration="3.04045979s" totalImagesPullingTime="1.483094024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:07:26.0394358 +0000 UTC m=+452.414439620" watchObservedRunningTime="2026-07-09 04:07:26.04045979 +0000 UTC m=+452.415463611"
Jul 09 04:07:30.988959 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:30.988921    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-2l5z7"
Jul 09 04:07:37.024480 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:07:37.024444    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-765cd6b57c-lcvtc"
Jul 09 04:08:42.149561 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.149530    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6887f95f6-298l5"]
Jul 09 04:08:42.152086 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.152064    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.167882 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.167854    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6887f95f6-298l5"]
Jul 09 04:08:42.239440 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.239408    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88b9\" (UniqueName: \"kubernetes.io/projected/f52501a6-04a1-4a4f-af27-b87d91f49946-kube-api-access-l88b9\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.239440 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.239444    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-service-ca\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.239665 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.239463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-trusted-ca-bundle\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.239665 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.239486    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-oauth-serving-cert\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.239665 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.239564    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f52501a6-04a1-4a4f-af27-b87d91f49946-console-serving-cert\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.239665 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.239605    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-console-config\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.239665 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.239640    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f52501a6-04a1-4a4f-af27-b87d91f49946-console-oauth-config\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.340252 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.340186    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-console-config\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.340444 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.340273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f52501a6-04a1-4a4f-af27-b87d91f49946-console-oauth-config\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.340444 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.340361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88b9\" (UniqueName: \"kubernetes.io/projected/f52501a6-04a1-4a4f-af27-b87d91f49946-kube-api-access-l88b9\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.340444 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.340388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-service-ca\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.340444 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.340410    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-trusted-ca-bundle\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.340659 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.340467    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-oauth-serving-cert\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.340659 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.340512    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f52501a6-04a1-4a4f-af27-b87d91f49946-console-serving-cert\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.340988 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.340959    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-console-config\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.341117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.341068    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-service-ca\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.341307 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.341279    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-oauth-serving-cert\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.341498 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.341477    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f52501a6-04a1-4a4f-af27-b87d91f49946-trusted-ca-bundle\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.342984 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.342955    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f52501a6-04a1-4a4f-af27-b87d91f49946-console-oauth-config\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.343200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.343183    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f52501a6-04a1-4a4f-af27-b87d91f49946-console-serving-cert\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.348118 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.348094    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88b9\" (UniqueName: \"kubernetes.io/projected/f52501a6-04a1-4a4f-af27-b87d91f49946-kube-api-access-l88b9\") pod \"console-6887f95f6-298l5\" (UID: \"f52501a6-04a1-4a4f-af27-b87d91f49946\") " pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.463369 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.463286    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:42.590353 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:42.590320    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6887f95f6-298l5"]
Jul 09 04:08:42.594960 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:08:42.594932    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52501a6_04a1_4a4f_af27_b87d91f49946.slice/crio-44cf55597fd35496d8e63ef9e2441836a71ba64f181e25afd176f0a36b1401a7 WatchSource:0}: Error finding container 44cf55597fd35496d8e63ef9e2441836a71ba64f181e25afd176f0a36b1401a7: Status 404 returned error can't find the container with id 44cf55597fd35496d8e63ef9e2441836a71ba64f181e25afd176f0a36b1401a7
Jul 09 04:08:43.351487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:43.351443    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6887f95f6-298l5" event={"ID":"f52501a6-04a1-4a4f-af27-b87d91f49946","Type":"ContainerStarted","Data":"3c75c6e6c16c74dd7c18b6e9c9d7970531b4ca0a228c448d3d79f05863f95f9d"}
Jul 09 04:08:43.351888 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:43.351528    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6887f95f6-298l5" event={"ID":"f52501a6-04a1-4a4f-af27-b87d91f49946","Type":"ContainerStarted","Data":"44cf55597fd35496d8e63ef9e2441836a71ba64f181e25afd176f0a36b1401a7"}
Jul 09 04:08:43.369838 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:43.369789    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6887f95f6-298l5" podStartSLOduration=1.369774933 podStartE2EDuration="1.369774933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:08:43.367516715 +0000 UTC m=+529.742520534" watchObservedRunningTime="2026-07-09 04:08:43.369774933 +0000 UTC m=+529.744778750"
Jul 09 04:08:52.463782 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:52.463704    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:52.463782 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:52.463767    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:52.468290 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:52.468270    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:53.399580 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:53.399556    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6887f95f6-298l5"
Jul 09 04:08:53.455800 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:08:53.455759    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c7888d595-p59g6"]
Jul 09 04:09:06.061051 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.061016    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"]
Jul 09 04:09:06.063334 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.063316    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.065938 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.065906    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 04:09:06.066050 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.065971    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 04:09:06.066050 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.066015    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 09 04:09:06.067002 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.066986    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 09 04:09:06.067059 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.067007    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lvckb\""
Jul 09 04:09:06.072087 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.072065    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"]
Jul 09 04:09:06.140931 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.140896    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3b76c957-042f-4545-b948-23071e7bc1ce-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-42zx9\" (UID: \"3b76c957-042f-4545-b948-23071e7bc1ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.141113 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.140981    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfw2w\" (UniqueName: \"kubernetes.io/projected/3b76c957-042f-4545-b948-23071e7bc1ce-kube-api-access-bfw2w\") pod \"kuadrant-console-plugin-66b5774985-42zx9\" (UID: \"3b76c957-042f-4545-b948-23071e7bc1ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.141113 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.141048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b76c957-042f-4545-b948-23071e7bc1ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-42zx9\" (UID: \"3b76c957-042f-4545-b948-23071e7bc1ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.241557 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.241516    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfw2w\" (UniqueName: \"kubernetes.io/projected/3b76c957-042f-4545-b948-23071e7bc1ce-kube-api-access-bfw2w\") pod \"kuadrant-console-plugin-66b5774985-42zx9\" (UID: \"3b76c957-042f-4545-b948-23071e7bc1ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.241750 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.241609    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b76c957-042f-4545-b948-23071e7bc1ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-42zx9\" (UID: \"3b76c957-042f-4545-b948-23071e7bc1ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.241750 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.241655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3b76c957-042f-4545-b948-23071e7bc1ce-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-42zx9\" (UID: \"3b76c957-042f-4545-b948-23071e7bc1ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.242288 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.242269    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3b76c957-042f-4545-b948-23071e7bc1ce-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-42zx9\" (UID: \"3b76c957-042f-4545-b948-23071e7bc1ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.244089 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.244066    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b76c957-042f-4545-b948-23071e7bc1ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-42zx9\" (UID: \"3b76c957-042f-4545-b948-23071e7bc1ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.249502 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.249484    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfw2w\" (UniqueName: \"kubernetes.io/projected/3b76c957-042f-4545-b948-23071e7bc1ce-kube-api-access-bfw2w\") pod \"kuadrant-console-plugin-66b5774985-42zx9\" (UID: \"3b76c957-042f-4545-b948-23071e7bc1ce\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.373268 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.373159    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"
Jul 09 04:09:06.504395 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:06.504371    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9"]
Jul 09 04:09:06.507305 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:09:06.507279    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b76c957_042f_4545_b948_23071e7bc1ce.slice/crio-5d0703f8fd6fa6cb90289eaeb86134e7745696305a7b8bdc031001c6fb6d1ca6 WatchSource:0}: Error finding container 5d0703f8fd6fa6cb90289eaeb86134e7745696305a7b8bdc031001c6fb6d1ca6: Status 404 returned error can't find the container with id 5d0703f8fd6fa6cb90289eaeb86134e7745696305a7b8bdc031001c6fb6d1ca6
Jul 09 04:09:07.461860 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:07.461827    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9" event={"ID":"3b76c957-042f-4545-b948-23071e7bc1ce","Type":"ContainerStarted","Data":"5d0703f8fd6fa6cb90289eaeb86134e7745696305a7b8bdc031001c6fb6d1ca6"}
Jul 09 04:09:18.479195 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:18.479148    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7c7888d595-p59g6" podUID="45bdd57c-753f-4749-9d28-955b3d92a146" containerName="console" containerID="cri-o://a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc" gracePeriod=15
Jul 09 04:09:26.498472 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:26.498428    2558 patch_prober.go:28] interesting pod/console-7c7888d595-p59g6 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.21:8443/health\": dial tcp 10.133.0.21:8443: connect: connection refused" start-of-body=
Jul 09 04:09:26.498868 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:26.498514    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7c7888d595-p59g6" podUID="45bdd57c-753f-4749-9d28-955b3d92a146" containerName="console" probeResult="failure" output="Get \"https://10.133.0.21:8443/health\": dial tcp 10.133.0.21:8443: connect: connection refused"
Jul 09 04:09:29.774939 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.774913    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:09:29.866556 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.866533    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-oauth-serving-cert\") pod \"45bdd57c-753f-4749-9d28-955b3d92a146\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") "
Jul 09 04:09:29.866683 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.866562    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntxtb\" (UniqueName: \"kubernetes.io/projected/45bdd57c-753f-4749-9d28-955b3d92a146-kube-api-access-ntxtb\") pod \"45bdd57c-753f-4749-9d28-955b3d92a146\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") "
Jul 09 04:09:29.866683 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.866578    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-trusted-ca-bundle\") pod \"45bdd57c-753f-4749-9d28-955b3d92a146\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") "
Jul 09 04:09:29.866683 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.866617    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-console-config\") pod \"45bdd57c-753f-4749-9d28-955b3d92a146\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") "
Jul 09 04:09:29.866683 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.866646    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-service-ca\") pod \"45bdd57c-753f-4749-9d28-955b3d92a146\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") "
Jul 09 04:09:29.866894 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.866683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-oauth-config\") pod \"45bdd57c-753f-4749-9d28-955b3d92a146\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") "
Jul 09 04:09:29.866894 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.866708    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-serving-cert\") pod \"45bdd57c-753f-4749-9d28-955b3d92a146\" (UID: \"45bdd57c-753f-4749-9d28-955b3d92a146\") "
Jul 09 04:09:29.867000 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.866941    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "45bdd57c-753f-4749-9d28-955b3d92a146" (UID: "45bdd57c-753f-4749-9d28-955b3d92a146"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:09:29.867060 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.866995    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "45bdd57c-753f-4749-9d28-955b3d92a146" (UID: "45bdd57c-753f-4749-9d28-955b3d92a146"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:09:29.867378 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.867339    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-service-ca" (OuterVolumeSpecName: "service-ca") pod "45bdd57c-753f-4749-9d28-955b3d92a146" (UID: "45bdd57c-753f-4749-9d28-955b3d92a146"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:09:29.867378 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.867345    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-console-config" (OuterVolumeSpecName: "console-config") pod "45bdd57c-753f-4749-9d28-955b3d92a146" (UID: "45bdd57c-753f-4749-9d28-955b3d92a146"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 04:09:29.868991 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.868968    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "45bdd57c-753f-4749-9d28-955b3d92a146" (UID: "45bdd57c-753f-4749-9d28-955b3d92a146"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:09:29.869366 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.869348    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bdd57c-753f-4749-9d28-955b3d92a146-kube-api-access-ntxtb" (OuterVolumeSpecName: "kube-api-access-ntxtb") pod "45bdd57c-753f-4749-9d28-955b3d92a146" (UID: "45bdd57c-753f-4749-9d28-955b3d92a146"). InnerVolumeSpecName "kube-api-access-ntxtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:09:29.869497 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.869469    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "45bdd57c-753f-4749-9d28-955b3d92a146" (UID: "45bdd57c-753f-4749-9d28-955b3d92a146"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:09:29.968195 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.968167    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-console-config\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:09:29.968195 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.968193    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-service-ca\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:09:29.968417 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.968243    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-oauth-config\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:09:29.968417 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.968258    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45bdd57c-753f-4749-9d28-955b3d92a146-console-serving-cert\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:09:29.968417 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.968270    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-oauth-serving-cert\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:09:29.968417 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.968281    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntxtb\" (UniqueName: \"kubernetes.io/projected/45bdd57c-753f-4749-9d28-955b3d92a146-kube-api-access-ntxtb\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:09:29.968417 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:29.968292    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45bdd57c-753f-4749-9d28-955b3d92a146-trusted-ca-bundle\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:09:30.579204 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.579169    2558 generic.go:356] "Generic (PLEG): container finished" podID="45bdd57c-753f-4749-9d28-955b3d92a146" containerID="a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc" exitCode=0
Jul 09 04:09:30.579390 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.579230    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c7888d595-p59g6"
Jul 09 04:09:30.579390 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.579254    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7888d595-p59g6" event={"ID":"45bdd57c-753f-4749-9d28-955b3d92a146","Type":"ContainerDied","Data":"a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc"}
Jul 09 04:09:30.579390 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.579307    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c7888d595-p59g6" event={"ID":"45bdd57c-753f-4749-9d28-955b3d92a146","Type":"ContainerDied","Data":"2549c635461d4f6980c147f98d20cc48255a10864aca3c78a4a6930f96cf36d4"}
Jul 09 04:09:30.579390 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.579334    2558 scope.go:117] "RemoveContainer" containerID="a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc"
Jul 09 04:09:30.580813 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.580790    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9" event={"ID":"3b76c957-042f-4545-b948-23071e7bc1ce","Type":"ContainerStarted","Data":"32746909ad60b4a088bdf773e0a15797e21825306823c6d95e2fee32375cbe90"}
Jul 09 04:09:30.588018 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.587998    2558 scope.go:117] "RemoveContainer" containerID="a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc"
Jul 09 04:09:30.588294 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:09:30.588273    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc\": container with ID starting with a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc not found: ID does not exist" containerID="a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc"
Jul 09 04:09:30.588349 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.588303    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc"} err="failed to get container status \"a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc\": rpc error: code = NotFound desc = could not find container \"a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc\": container with ID starting with a212ad6b6ab08470600d2cc90fbe1c6b07ce65d18905ec29c1813260edb210fc not found: ID does not exist"
Jul 09 04:09:30.596746 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.596710    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c7888d595-p59g6"]
Jul 09 04:09:30.604141 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:30.604120    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c7888d595-p59g6"]
Jul 09 04:09:32.109489 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:09:32.109454    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45bdd57c-753f-4749-9d28-955b3d92a146" path="/var/lib/kubelet/pods/45bdd57c-753f-4749-9d28-955b3d92a146/volumes"
Jul 09 04:10:00.411128 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.411071    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-42zx9" podStartSLOduration=31.062133871 podStartE2EDuration="54.41105385s" totalImagesPullingTime="23.348919979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:09:30.616779687 +0000 UTC m=+576.991783529" watchObservedRunningTime="2026-07-09 04:10:00.41105385 +0000 UTC m=+606.786057668"
Jul 09 04:10:00.412596 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.412577    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:10:00.412895 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.412884    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45bdd57c-753f-4749-9d28-955b3d92a146" containerName="console"
Jul 09 04:10:00.412941 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.412897    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45bdd57c-753f-4749-9d28-955b3d92a146" containerName="console"
Jul 09 04:10:00.412991 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.412960    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45bdd57c-753f-4749-9d28-955b3d92a146" containerName="console"
Jul 09 04:10:00.530199 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.530161    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:10:00.530377 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.530247    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:10:00.530377 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.530348    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:10:00.532963 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.532942    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 04:10:00.629465 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.629427    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbfw\" (UniqueName: \"kubernetes.io/projected/a8e8a32c-9a5f-4b18-ac4b-e451f4f98399-kube-api-access-mzbfw\") pod \"limitador-limitador-54d6d988d5-9mrbc\" (UID: \"a8e8a32c-9a5f-4b18-ac4b-e451f4f98399\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:10:00.629629 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.629473    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a8e8a32c-9a5f-4b18-ac4b-e451f4f98399-config-file\") pod \"limitador-limitador-54d6d988d5-9mrbc\" (UID: \"a8e8a32c-9a5f-4b18-ac4b-e451f4f98399\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:10:00.730671 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.730593    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbfw\" (UniqueName: \"kubernetes.io/projected/a8e8a32c-9a5f-4b18-ac4b-e451f4f98399-kube-api-access-mzbfw\") pod \"limitador-limitador-54d6d988d5-9mrbc\" (UID: \"a8e8a32c-9a5f-4b18-ac4b-e451f4f98399\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:10:00.730671 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.730640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a8e8a32c-9a5f-4b18-ac4b-e451f4f98399-config-file\") pod \"limitador-limitador-54d6d988d5-9mrbc\" (UID: \"a8e8a32c-9a5f-4b18-ac4b-e451f4f98399\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:10:00.731296 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.731277    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a8e8a32c-9a5f-4b18-ac4b-e451f4f98399-config-file\") pod \"limitador-limitador-54d6d988d5-9mrbc\" (UID: \"a8e8a32c-9a5f-4b18-ac4b-e451f4f98399\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:10:00.739363 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.739333    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbfw\" (UniqueName: \"kubernetes.io/projected/a8e8a32c-9a5f-4b18-ac4b-e451f4f98399-kube-api-access-mzbfw\") pod \"limitador-limitador-54d6d988d5-9mrbc\" (UID: \"a8e8a32c-9a5f-4b18-ac4b-e451f4f98399\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:10:00.840167 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:00.840127    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:10:01.069362 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:01.069337    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:10:01.070956 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:10:01.070928    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e8a32c_9a5f_4b18_ac4b_e451f4f98399.slice/crio-2ac8a07b5ccea3045a963705575d78b77b3099da23292864781f8b79bb1e4ccd WatchSource:0}: Error finding container 2ac8a07b5ccea3045a963705575d78b77b3099da23292864781f8b79bb1e4ccd: Status 404 returned error can't find the container with id 2ac8a07b5ccea3045a963705575d78b77b3099da23292864781f8b79bb1e4ccd
Jul 09 04:10:01.719591 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:01.719545    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc" event={"ID":"a8e8a32c-9a5f-4b18-ac4b-e451f4f98399","Type":"ContainerStarted","Data":"2ac8a07b5ccea3045a963705575d78b77b3099da23292864781f8b79bb1e4ccd"}
Jul 09 04:10:04.735088 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:04.735050    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc" event={"ID":"a8e8a32c-9a5f-4b18-ac4b-e451f4f98399","Type":"ContainerStarted","Data":"b8959d3685c6a388791bc5e4a6fad54f1e9c0ea7d17ffb8c294c46856d924584"}
Jul 09 04:10:04.735510 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:04.735198    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:10:04.751288 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:04.751247    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc" podStartSLOduration=2.001628098 podStartE2EDuration="4.751233688s" totalImagesPullingTime="2.74960559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:10:04.750664658 +0000 UTC m=+611.125668477" watchObservedRunningTime="2026-07-09 04:10:04.751233688 +0000 UTC m=+611.126237506"
Jul 09 04:10:15.741089 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:10:15.741058    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"
Jul 09 04:11:11.837842 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:11.837806    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:11:31.783792 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:31.783755    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:11:37.489819 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:37.489779    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:11:49.876388 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:49.876301    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:11:54.681953 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.681920    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"]
Jul 09 04:11:54.684276 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.684253    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.688435 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.688412    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 09 04:11:54.688564 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.688414    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 09 04:11:54.688564 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.688415    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-trvcm\""
Jul 09 04:11:54.688673 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.688467    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 09 04:11:54.694728 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.694695    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"]
Jul 09 04:11:54.804613 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.804563    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.804613 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.804610    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpnc\" (UniqueName: \"kubernetes.io/projected/a3521141-c52e-41c5-a9c6-0645a65cf678-kube-api-access-twpnc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.804850 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.804633    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.804850 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.804697    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.804850 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.804733    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3521141-c52e-41c5-a9c6-0645a65cf678-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.804850 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.804762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.905983 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.905931    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.905983 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.905985    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpnc\" (UniqueName: \"kubernetes.io/projected/a3521141-c52e-41c5-a9c6-0645a65cf678-kube-api-access-twpnc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.906233 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.906115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.906233 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.906165    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.906233 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.906195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3521141-c52e-41c5-a9c6-0645a65cf678-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.906395 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.906249    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.906562 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.906534    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.906562 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.906559    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.906714 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.906599    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.908092 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.908065    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3521141-c52e-41c5-a9c6-0645a65cf678-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.908513 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.908497    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3521141-c52e-41c5-a9c6-0645a65cf678-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.913380 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.913363    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpnc\" (UniqueName: \"kubernetes.io/projected/a3521141-c52e-41c5-a9c6-0645a65cf678-kube-api-access-twpnc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx\" (UID: \"a3521141-c52e-41c5-a9c6-0645a65cf678\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:54.999436 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:54.999350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:11:55.127955 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:55.127909    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"]
Jul 09 04:11:55.130348 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:11:55.130308    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3521141_c52e_41c5_a9c6_0645a65cf678.slice/crio-c6ba42a89f5c6a97256f704731ad6dfd22db780ab1fa6f0ed178ae51bd323a45 WatchSource:0}: Error finding container c6ba42a89f5c6a97256f704731ad6dfd22db780ab1fa6f0ed178ae51bd323a45: Status 404 returned error can't find the container with id c6ba42a89f5c6a97256f704731ad6dfd22db780ab1fa6f0ed178ae51bd323a45
Jul 09 04:11:55.131604 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:55.131587    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 04:11:55.252376 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:55.252295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx" event={"ID":"a3521141-c52e-41c5-a9c6-0645a65cf678","Type":"ContainerStarted","Data":"c6ba42a89f5c6a97256f704731ad6dfd22db780ab1fa6f0ed178ae51bd323a45"}
Jul 09 04:11:56.260157 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:56.260123    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx" event={"ID":"a3521141-c52e-41c5-a9c6-0645a65cf678","Type":"ContainerStarted","Data":"c246809ec20096ce87e10672607b00c4348562b19f7b4b44fb0ba5da7acce842"}
Jul 09 04:11:56.277881 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:56.277828    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx" podStartSLOduration=1.287511455 podStartE2EDuration="2.277812615s" totalImagesPullingTime="990.30116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:11:56.276281117 +0000 UTC m=+722.651284961" watchObservedRunningTime="2026-07-09 04:11:56.277812615 +0000 UTC m=+722.652816474"
Jul 09 04:11:56.389437 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:11:56.389340    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:12:04.999490 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:04.999451    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:12:04.999940 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:04.999517    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:12:05.011830 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:05.011798    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:12:05.313673 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:05.313594    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx"
Jul 09 04:12:09.488009 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:09.487973    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:12:14.383916 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.383873    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"]
Jul 09 04:12:14.386936 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.386916    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.389990 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.389963    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 09 04:12:14.397890 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.397860    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"]
Jul 09 04:12:14.474627 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.474585    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p6kx\" (UniqueName: \"kubernetes.io/projected/56b63dba-264b-460c-808f-54af0557d23c-kube-api-access-9p6kx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.474780 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.474639    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b63dba-264b-460c-808f-54af0557d23c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.474780 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.474724    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.474780 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.474771    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.474917 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.474793    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.474917 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.474846    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.575620 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.575568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b63dba-264b-460c-808f-54af0557d23c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.575830 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.575644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.575830 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.575690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.575830 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.575714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.575830 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.575738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.575830 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.575799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6kx\" (UniqueName: \"kubernetes.io/projected/56b63dba-264b-460c-808f-54af0557d23c-kube-api-access-9p6kx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.576167 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.576131    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.576167 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.576157    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.576354 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.576202    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.578025 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.577992    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b63dba-264b-460c-808f-54af0557d23c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.578172 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.578156    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b63dba-264b-460c-808f-54af0557d23c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.584270 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.584246    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p6kx\" (UniqueName: \"kubernetes.io/projected/56b63dba-264b-460c-808f-54af0557d23c-kube-api-access-9p6kx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6\" (UID: \"56b63dba-264b-460c-808f-54af0557d23c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.702029 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.701985    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:14.835398 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:14.835372    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"]
Jul 09 04:12:14.837682 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:12:14.837654    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56b63dba_264b_460c_808f_54af0557d23c.slice/crio-74e664db808242bdb23e7c9551a96d21553cef04fb73460102723e0d30c29333 WatchSource:0}: Error finding container 74e664db808242bdb23e7c9551a96d21553cef04fb73460102723e0d30c29333: Status 404 returned error can't find the container with id 74e664db808242bdb23e7c9551a96d21553cef04fb73460102723e0d30c29333
Jul 09 04:12:15.348499 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:15.348410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6" event={"ID":"56b63dba-264b-460c-808f-54af0557d23c","Type":"ContainerStarted","Data":"50d47ac88e2f12b9eff20e539b85df399fe8331ffe67741691bd3b6a038cc7e5"}
Jul 09 04:12:15.348499 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:15.348457    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6" event={"ID":"56b63dba-264b-460c-808f-54af0557d23c","Type":"ContainerStarted","Data":"74e664db808242bdb23e7c9551a96d21553cef04fb73460102723e0d30c29333"}
Jul 09 04:12:15.378836 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:15.378787    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6" podStartSLOduration=1.20398801 podStartE2EDuration="1.378769925s" totalImagesPullingTime="174.781915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:12:15.377715497 +0000 UTC m=+741.752719317" watchObservedRunningTime="2026-07-09 04:12:15.378769925 +0000 UTC m=+741.753773721"
Jul 09 04:12:16.480824 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:16.480789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:12:24.702760 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:24.702725    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:24.703379 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:24.702778    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:24.715052 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:24.715029    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:12:25.407312 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:12:25.407284    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6"
Jul 09 04:14:13.395564 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:14:13.395533    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:14:23.789767 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:14:23.789736    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:14:34.488999 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:14:34.488964    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:14:43.583828 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:14:43.583785    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:14:54.579170 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:14:54.579087    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:15:20.485068 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:15:20.485028    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:15:32.084806 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:15:32.084769    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:16:04.379242 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:16:04.379193    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:16:12.177501 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:16:12.177465    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:16:22.784204 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:16:22.784123    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:16:30.679818 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:16:30.679781    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:16:40.584918 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:16:40.584879    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:16:51.783688 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:16:51.783654    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:17:03.184518 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:17:03.184480    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:17:13.483360 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:17:13.483320    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:17:35.083527 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:17:35.083485    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:17:44.179811 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:17:44.179777    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:18:06.696525 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:18:06.696436    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:18:17.288130 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:18:17.288094    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:18:33.996014 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:18:33.995975    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:18:43.389133 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:18:43.389095    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:18:52.992593 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:18:52.992557    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:19:03.087871 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:19:03.087830    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:19:12.883263 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:19:12.883226    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:20:58.784971 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:20:58.784894    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:21:07.490428 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:21:07.490393    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:21:17.981407 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:21:17.981367    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:21:29.782723 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:21:29.782686    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:21:39.382515 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:21:39.382475    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:21:50.985340 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:21:50.985307    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:21:59.582396 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:21:59.582361    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:22:10.989561 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:22:10.989528    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:22:21.685716 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:22:21.685632    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:22:30.281567 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:22:30.281529    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:22:41.243196 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:22:41.243161    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:23:02.685388 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:23:02.685354    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:23:11.685822 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:23:11.685789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:23:21.784263 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:23:21.784227    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:23:32.576943 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:23:32.576903    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:23:39.688541 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:23:39.688507    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:23:52.585140 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:23:52.585062    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:24:06.688055 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:24:06.688019    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:24:16.086546 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:24:16.086506    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:24:29.283486 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:24:29.283446    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:24:38.384101 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:24:38.384061    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:24:53.689329 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:24:53.689290    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:25:07.847471 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:25:07.847432    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:25:17.483937 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:25:17.483858    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:25:26.293508 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:25:26.293464    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:25:37.986781 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:25:37.986741    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:25:55.377514 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:25:55.377474    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:26:07.387898 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:26:07.387863    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:26:20.983418 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:26:20.983385    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:26:33.696127 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:26:33.696093    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:26:43.077559 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:26:43.077521    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:26:52.294274 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:26:52.294180    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:27:10.786945 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:27:10.786908    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:27:20.480578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:27:20.480538    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:27:28.283148 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:27:28.283111    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:27:39.789592 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:27:39.789555    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:28:58.581978 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:28:58.581943    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:29:07.884569 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:29:07.884535    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:29:19.186709 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:29:19.186669    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:29:29.891867 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:29:29.891829    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:30:00.580372 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:00.580333    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:30:11.701947 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.701910    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"]
Jul 09 04:30:11.704487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.704468    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.707402 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.707380    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e747c320\"/\"default-dockercfg-2t7v6\""
Jul 09 04:30:11.707515 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.707445    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e747c320\"/\"kube-root-ca.crt\""
Jul 09 04:30:11.707515 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.707462    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e747c320\"/\"openshift-service-ca.crt\""
Jul 09 04:30:11.708737 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.708723    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e747c320\"/\"e2e-lifecycle-model-e747c320-kserve-self-signed-certs\""
Jul 09 04:30:11.713931 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.713908    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"]
Jul 09 04:30:11.870690 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.870654    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-model-cache\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.870888 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.870704    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2zgq\" (UniqueName: \"kubernetes.io/projected/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-kube-api-access-d2zgq\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.870888 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.870767    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-dshm\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.870888 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.870795    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tmp-dir\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.870888 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.870846    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tls-certs\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.871056 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.870920    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-home\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.972345 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.972253    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tls-certs\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.972345 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.972314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-home\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.972345 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.972341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-model-cache\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.972626 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.972365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2zgq\" (UniqueName: \"kubernetes.io/projected/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-kube-api-access-d2zgq\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.972626 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.972395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-dshm\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.972626 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.972419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tmp-dir\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.972860 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.972837    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-model-cache\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.972900 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.972847    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-home\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.972900 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.972888    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tmp-dir\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.974608 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.974588    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-dshm\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.974958 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.974938    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tls-certs\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:11.980227 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:11.980186    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2zgq\" (UniqueName: \"kubernetes.io/projected/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-kube-api-access-d2zgq\") pod \"e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") " pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:12.015098 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:12.015067    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:12.144289 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:12.144265    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"]
Jul 09 04:30:12.146436 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:30:12.146410    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod516e104b_f1d4_4b89_a5b8_e64ce7bbe3c3.slice/crio-c16de97ce54f4be87e4b2f3e7c67dcd0f65325b6cb1ec1162a85276222e46d81 WatchSource:0}: Error finding container c16de97ce54f4be87e4b2f3e7c67dcd0f65325b6cb1ec1162a85276222e46d81: Status 404 returned error can't find the container with id c16de97ce54f4be87e4b2f3e7c67dcd0f65325b6cb1ec1162a85276222e46d81
Jul 09 04:30:12.220526 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:12.220497    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6" event={"ID":"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3","Type":"ContainerStarted","Data":"0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015"}
Jul 09 04:30:12.220685 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:12.220536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6" event={"ID":"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3","Type":"ContainerStarted","Data":"c16de97ce54f4be87e4b2f3e7c67dcd0f65325b6cb1ec1162a85276222e46d81"}
Jul 09 04:30:12.238194 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:12.238105    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6" podStartSLOduration=1.238088909 podStartE2EDuration="1.238088909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:30:12.237443474 +0000 UTC m=+1818.612447293" watchObservedRunningTime="2026-07-09 04:30:12.238088909 +0000 UTC m=+1818.613092768"
Jul 09 04:30:15.487805 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:15.487769    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:30:22.015917 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:22.015872    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:22.016321 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:22.015941    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:22.028586 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:22.028559    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:22.277426 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:22.277346    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:29.290926 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:29.290886    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:30:42.659262 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:42.659223    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"]
Jul 09 04:30:44.384482 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:44.384448    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:30:57.681532 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:57.681483    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6" podUID="516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" containerName="main" containerID="cri-o://0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015" gracePeriod=45
Jul 09 04:30:57.934769 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:57.934708    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:58.070701 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.070662    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tmp-dir\") pod \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") "
Jul 09 04:30:58.070882 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.070730    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-model-cache\") pod \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") "
Jul 09 04:30:58.070882 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.070751    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-dshm\") pod \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") "
Jul 09 04:30:58.070882 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.070773    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tls-certs\") pod \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") "
Jul 09 04:30:58.070882 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.070799    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-home\") pod \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") "
Jul 09 04:30:58.070882 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.070829    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2zgq\" (UniqueName: \"kubernetes.io/projected/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-kube-api-access-d2zgq\") pod \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\" (UID: \"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3\") "
Jul 09 04:30:58.071139 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.071013    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" (UID: "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:30:58.071139 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.071064    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" (UID: "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:30:58.071139 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.071095    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-home" (OuterVolumeSpecName: "home") pod "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" (UID: "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:30:58.072995 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.072969    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-dshm" (OuterVolumeSpecName: "dshm") pod "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" (UID: "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:30:58.073095 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.073021    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" (UID: "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:30:58.073095 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.073034    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-kube-api-access-d2zgq" (OuterVolumeSpecName: "kube-api-access-d2zgq") pod "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" (UID: "516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3"). InnerVolumeSpecName "kube-api-access-d2zgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:30:58.172433 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.172401    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tmp-dir\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:30:58.172433 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.172430    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-model-cache\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:30:58.172433 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.172440    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-dshm\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:30:58.172656 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.172448    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-tls-certs\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:30:58.172656 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.172455    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-home\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:30:58.172656 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.172463    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2zgq\" (UniqueName: \"kubernetes.io/projected/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3-kube-api-access-d2zgq\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:30:58.434357 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.434316    2558 generic.go:356] "Generic (PLEG): container finished" podID="516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" containerID="0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015" exitCode=0
Jul 09 04:30:58.434524 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.434383    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"
Jul 09 04:30:58.434524 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.434383    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6" event={"ID":"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3","Type":"ContainerDied","Data":"0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015"}
Jul 09 04:30:58.434524 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.434491    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6" event={"ID":"516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3","Type":"ContainerDied","Data":"c16de97ce54f4be87e4b2f3e7c67dcd0f65325b6cb1ec1162a85276222e46d81"}
Jul 09 04:30:58.434524 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.434512    2558 scope.go:117] "RemoveContainer" containerID="0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015"
Jul 09 04:30:58.442940 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.442922    2558 scope.go:117] "RemoveContainer" containerID="0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015"
Jul 09 04:30:58.443174 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:30:58.443156    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015\": container with ID starting with 0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015 not found: ID does not exist" containerID="0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015"
Jul 09 04:30:58.443320 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.443183    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015"} err="failed to get container status \"0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015\": rpc error: code = NotFound desc = could not find container \"0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015\": container with ID starting with 0714d95e2c2f9da9c1cd7d1ccbae47c6733fd6df8e6aea7e72a213d024fd7015 not found: ID does not exist"
Jul 09 04:30:58.449615 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.449591    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"]
Jul 09 04:30:58.453308 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:30:58.453288    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e747c320/e2e-lifecycle-model-e747c320-kserve-fd5d5bff9-nthq6"]
Jul 09 04:31:00.109660 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:31:00.109627    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" path="/var/lib/kubelet/pods/516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3/volumes"
Jul 09 04:31:21.984167 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:31:21.984133    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:31:31.784348 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:31:31.784314    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:31:42.287817 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:31:42.287777    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:32:30.381099 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:30.381057    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:32:43.477819 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.477786    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"]
Jul 09 04:32:43.478391 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.478325    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" containerName="main"
Jul 09 04:32:43.478391 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.478345    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" containerName="main"
Jul 09 04:32:43.478510 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.478448    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="516e104b-f1d4-4b89-a5b8-e64ce7bbe3c3" containerName="main"
Jul 09 04:32:43.481479 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.481459    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.485117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.485080    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c184fc\"/\"test-model-c184fc-kserve-self-signed-certs\""
Jul 09 04:32:43.485117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.485111    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c184fc\"/\"kube-root-ca.crt\""
Jul 09 04:32:43.485342 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.485149    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c184fc\"/\"openshift-service-ca.crt\""
Jul 09 04:32:43.485342 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.485116    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c184fc\"/\"default-dockercfg-b5bpz\""
Jul 09 04:32:43.490029 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.489995    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"]
Jul 09 04:32:43.581753 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.581715    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h475w\" (UniqueName: \"kubernetes.io/projected/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-kube-api-access-h475w\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.581753 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.581756    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-home\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.582003 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.581779    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-dshm\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.582003 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.581849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tmp-dir\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.582003 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.581963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tls-certs\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.582129 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.582012    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-model-cache\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.583217 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.583188    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:32:43.682540 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.682508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-home\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.682540 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.682544    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-dshm\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.682783 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.682570    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tmp-dir\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.682783 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.682608    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tls-certs\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.682783 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.682630    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-model-cache\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.682783 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.682694    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h475w\" (UniqueName: \"kubernetes.io/projected/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-kube-api-access-h475w\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.683001 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.682973    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-home\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.683177 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.683063    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tmp-dir\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.683378 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.683358    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-model-cache\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.685124 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.685097    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tls-certs\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.685201 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.685142    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-dshm\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.691023 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.691005    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h475w\" (UniqueName: \"kubernetes.io/projected/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-kube-api-access-h475w\") pod \"test-model-c184fc-kserve-9b6795595-zqlgk\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") " pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.792077 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.791986    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:43.931182 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:43.931155    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"]
Jul 09 04:32:43.933243 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:32:43.933220    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6f715a9_a4e0_46bd_ab31_3c6d565fff0f.slice/crio-a39ca268fea34475ab718fb1d67012ccb4030ac755faabfb5351d24c29939708 WatchSource:0}: Error finding container a39ca268fea34475ab718fb1d67012ccb4030ac755faabfb5351d24c29939708: Status 404 returned error can't find the container with id a39ca268fea34475ab718fb1d67012ccb4030ac755faabfb5351d24c29939708
Jul 09 04:32:44.925842 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:44.925802    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk" event={"ID":"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f","Type":"ContainerStarted","Data":"206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633"}
Jul 09 04:32:44.925842 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:44.925848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk" event={"ID":"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f","Type":"ContainerStarted","Data":"a39ca268fea34475ab718fb1d67012ccb4030ac755faabfb5351d24c29939708"}
Jul 09 04:32:44.945147 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:44.945090    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk" podStartSLOduration=1.9450748180000001 podStartE2EDuration="1.945074818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:32:44.944144973 +0000 UTC m=+1971.319148809" watchObservedRunningTime="2026-07-09 04:32:44.945074818 +0000 UTC m=+1971.320078636"
Jul 09 04:32:53.792687 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:53.792596    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:53.792687 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:53.792649    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:53.804878 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:53.804846    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:32:53.979818 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:32:53.979788    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:33:00.179919 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:00.179877    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:33:04.786719 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.786685    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"]
Jul 09 04:33:04.789900 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.789884    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.792567 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.792537    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b3e4d0\"/\"openshift-service-ca.crt\""
Jul 09 04:33:04.792695 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.792597    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b3e4d0\"/\"kube-root-ca.crt\""
Jul 09 04:33:04.793948 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.793930    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b3e4d0\"/\"default-dockercfg-nbjjb\""
Jul 09 04:33:04.794054 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.793962    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b3e4d0\"/\"test-model-b3e4d0-kserve-self-signed-certs\""
Jul 09 04:33:04.800140 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.800117    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"]
Jul 09 04:33:04.849622 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.849582    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66rvf\" (UniqueName: \"kubernetes.io/projected/5b53b508-7ce8-4320-8f8b-88cafc087c0b-kube-api-access-66rvf\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.849835 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.849644    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tmp-dir\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.849835 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.849680    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-home\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.849835 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.849699    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-dshm\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.849835 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.849734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tls-certs\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.849835 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.849783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-model-cache\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.950519 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.950483    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66rvf\" (UniqueName: \"kubernetes.io/projected/5b53b508-7ce8-4320-8f8b-88cafc087c0b-kube-api-access-66rvf\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.950708 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.950557    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tmp-dir\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.950708 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.950593    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-home\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.950708 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.950618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-dshm\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.950708 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.950660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tls-certs\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.950708 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.950698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-model-cache\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.951020 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.950997    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tmp-dir\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.951165 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.951066    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-model-cache\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.951375 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.951344    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-home\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.953230 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.953183    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-dshm\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.953987 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.953862    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tls-certs\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:04.960907 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:04.960878    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66rvf\" (UniqueName: \"kubernetes.io/projected/5b53b508-7ce8-4320-8f8b-88cafc087c0b-kube-api-access-66rvf\") pod \"test-model-b3e4d0-kserve-5bcdf598b5-xzbfj\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") " pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:05.100391 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:05.100307    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:05.233246 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:05.233189    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"]
Jul 09 04:33:05.235535 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:33:05.235506    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b53b508_7ce8_4320_8f8b_88cafc087c0b.slice/crio-e4fff5729f0458d2278a805010beea2a2ef6711004991793d9a9e9daa50e9b6e WatchSource:0}: Error finding container e4fff5729f0458d2278a805010beea2a2ef6711004991793d9a9e9daa50e9b6e: Status 404 returned error can't find the container with id e4fff5729f0458d2278a805010beea2a2ef6711004991793d9a9e9daa50e9b6e
Jul 09 04:33:06.033016 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:06.032978    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj" event={"ID":"5b53b508-7ce8-4320-8f8b-88cafc087c0b","Type":"ContainerStarted","Data":"e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113"}
Jul 09 04:33:06.033016 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:06.033022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj" event={"ID":"5b53b508-7ce8-4320-8f8b-88cafc087c0b","Type":"ContainerStarted","Data":"e4fff5729f0458d2278a805010beea2a2ef6711004991793d9a9e9daa50e9b6e"}
Jul 09 04:33:06.051621 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:06.051563    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj" podStartSLOduration=2.051546855 podStartE2EDuration="2.051546855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:33:06.049491935 +0000 UTC m=+1992.424495754" watchObservedRunningTime="2026-07-09 04:33:06.051546855 +0000 UTC m=+1992.426550672"
Jul 09 04:33:12.395600 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:12.395562    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:33:15.100810 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:15.100774    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:15.101274 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:15.100822    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:15.113352 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:15.113327    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:16.089919 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:16.089890    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:23.137040 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:23.136991    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"]
Jul 09 04:33:24.111409 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:24.111379    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"]
Jul 09 04:33:29.693943 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:29.693907    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:33:38.158229 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.158165    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk" podUID="a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" containerName="main" containerID="cri-o://206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633" gracePeriod=45
Jul 09 04:33:38.398186 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.398159    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:33:38.551317 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.551204    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-home\") pod \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") "
Jul 09 04:33:38.551317 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.551292    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tmp-dir\") pod \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") "
Jul 09 04:33:38.551551 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.551340    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h475w\" (UniqueName: \"kubernetes.io/projected/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-kube-api-access-h475w\") pod \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") "
Jul 09 04:33:38.551551 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.551380    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tls-certs\") pod \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") "
Jul 09 04:33:38.551551 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.551412    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-dshm\") pod \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") "
Jul 09 04:33:38.551551 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.551438    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-model-cache\") pod \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\" (UID: \"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f\") "
Jul 09 04:33:38.551551 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.551510    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-home" (OuterVolumeSpecName: "home") pod "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" (UID: "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:33:38.551800 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.551594    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" (UID: "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:33:38.551800 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.551781    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" (UID: "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:33:38.552031 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.552007    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-model-cache\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:38.552031 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.552034    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-home\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:38.552186 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.552047    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tmp-dir\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:38.553803 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.553769    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-dshm" (OuterVolumeSpecName: "dshm") pod "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" (UID: "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:33:38.553803 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.553784    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" (UID: "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:33:38.553947 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.553870    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-kube-api-access-h475w" (OuterVolumeSpecName: "kube-api-access-h475w") pod "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" (UID: "a6f715a9-a4e0-46bd-ab31-3c6d565fff0f"). InnerVolumeSpecName "kube-api-access-h475w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:33:38.653513 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.653465    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h475w\" (UniqueName: \"kubernetes.io/projected/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-kube-api-access-h475w\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:38.653513 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.653508    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-tls-certs\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:38.653513 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:38.653521    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f-dshm\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:39.132716 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.132651    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj" podUID="5b53b508-7ce8-4320-8f8b-88cafc087c0b" containerName="main" containerID="cri-o://e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113" gracePeriod=45
Jul 09 04:33:39.195097 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.195067    2558 generic.go:356] "Generic (PLEG): container finished" podID="a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" containerID="206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633" exitCode=0
Jul 09 04:33:39.195500 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.195108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk" event={"ID":"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f","Type":"ContainerDied","Data":"206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633"}
Jul 09 04:33:39.195500 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.195136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk" event={"ID":"a6f715a9-a4e0-46bd-ab31-3c6d565fff0f","Type":"ContainerDied","Data":"a39ca268fea34475ab718fb1d67012ccb4030ac755faabfb5351d24c29939708"}
Jul 09 04:33:39.195500 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.195112    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"
Jul 09 04:33:39.195500 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.195151    2558 scope.go:117] "RemoveContainer" containerID="206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633"
Jul 09 04:33:39.215259 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.215227    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"]
Jul 09 04:33:39.216722 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.216703    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c184fc/test-model-c184fc-kserve-9b6795595-zqlgk"]
Jul 09 04:33:39.243701 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.243679    2558 scope.go:117] "RemoveContainer" containerID="206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633"
Jul 09 04:33:39.244034 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:33:39.244017    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633\": container with ID starting with 206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633 not found: ID does not exist" containerID="206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633"
Jul 09 04:33:39.244102 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.244045    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633"} err="failed to get container status \"206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633\": rpc error: code = NotFound desc = could not find container \"206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633\": container with ID starting with 206c3fa2d49118172353b98d21721f22589356b87b14cc3f83c9455ad7dd9633 not found: ID does not exist"
Jul 09 04:33:39.389298 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.389240    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:39.461429 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.461394    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66rvf\" (UniqueName: \"kubernetes.io/projected/5b53b508-7ce8-4320-8f8b-88cafc087c0b-kube-api-access-66rvf\") pod \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") "
Jul 09 04:33:39.461578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.461451    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-dshm\") pod \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") "
Jul 09 04:33:39.461578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.461480    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tls-certs\") pod \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") "
Jul 09 04:33:39.461578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.461544    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tmp-dir\") pod \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") "
Jul 09 04:33:39.461578 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.461568    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-home\") pod \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") "
Jul 09 04:33:39.461812 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.461638    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-model-cache\") pod \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\" (UID: \"5b53b508-7ce8-4320-8f8b-88cafc087c0b\") "
Jul 09 04:33:39.461935 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.461907    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b53b508-7ce8-4320-8f8b-88cafc087c0b" (UID: "5b53b508-7ce8-4320-8f8b-88cafc087c0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:33:39.461989 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.461946    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-home" (OuterVolumeSpecName: "home") pod "5b53b508-7ce8-4320-8f8b-88cafc087c0b" (UID: "5b53b508-7ce8-4320-8f8b-88cafc087c0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:33:39.462049 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.462027    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b53b508-7ce8-4320-8f8b-88cafc087c0b" (UID: "5b53b508-7ce8-4320-8f8b-88cafc087c0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:33:39.463842 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.463816    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b53b508-7ce8-4320-8f8b-88cafc087c0b" (UID: "5b53b508-7ce8-4320-8f8b-88cafc087c0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:33:39.463997 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.463982    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b53b508-7ce8-4320-8f8b-88cafc087c0b-kube-api-access-66rvf" (OuterVolumeSpecName: "kube-api-access-66rvf") pod "5b53b508-7ce8-4320-8f8b-88cafc087c0b" (UID: "5b53b508-7ce8-4320-8f8b-88cafc087c0b"). InnerVolumeSpecName "kube-api-access-66rvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:33:39.463997 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.463988    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-dshm" (OuterVolumeSpecName: "dshm") pod "5b53b508-7ce8-4320-8f8b-88cafc087c0b" (UID: "5b53b508-7ce8-4320-8f8b-88cafc087c0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:33:39.563017 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.562983    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-model-cache\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:39.563017 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.563013    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66rvf\" (UniqueName: \"kubernetes.io/projected/5b53b508-7ce8-4320-8f8b-88cafc087c0b-kube-api-access-66rvf\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:39.563017 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.563022    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-dshm\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:39.563277 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.563031    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tls-certs\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:39.563277 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.563039    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-tmp-dir\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:39.563277 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:39.563046    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b53b508-7ce8-4320-8f8b-88cafc087c0b-home\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:33:40.110176 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.110143    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" path="/var/lib/kubelet/pods/a6f715a9-a4e0-46bd-ab31-3c6d565fff0f/volumes"
Jul 09 04:33:40.201151 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.201114    2558 generic.go:356] "Generic (PLEG): container finished" podID="5b53b508-7ce8-4320-8f8b-88cafc087c0b" containerID="e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113" exitCode=0
Jul 09 04:33:40.201598 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.201157    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"
Jul 09 04:33:40.201598 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.201222    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj" event={"ID":"5b53b508-7ce8-4320-8f8b-88cafc087c0b","Type":"ContainerDied","Data":"e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113"}
Jul 09 04:33:40.201598 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.201264    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj" event={"ID":"5b53b508-7ce8-4320-8f8b-88cafc087c0b","Type":"ContainerDied","Data":"e4fff5729f0458d2278a805010beea2a2ef6711004991793d9a9e9daa50e9b6e"}
Jul 09 04:33:40.201598 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.201286    2558 scope.go:117] "RemoveContainer" containerID="e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113"
Jul 09 04:33:40.209660 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.209638    2558 scope.go:117] "RemoveContainer" containerID="e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113"
Jul 09 04:33:40.209925 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:33:40.209904    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113\": container with ID starting with e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113 not found: ID does not exist" containerID="e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113"
Jul 09 04:33:40.209968 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.209934    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113"} err="failed to get container status \"e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113\": rpc error: code = NotFound desc = could not find container \"e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113\": container with ID starting with e33418a1b949c187793be8794c60932ff0c716093512e379ab6d04c5af05e113 not found: ID does not exist"
Jul 09 04:33:40.217217 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.217186    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"]
Jul 09 04:33:40.220379 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:40.220358    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b3e4d0/test-model-b3e4d0-kserve-5bcdf598b5-xzbfj"]
Jul 09 04:33:42.110877 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:33:42.110844    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b53b508-7ce8-4320-8f8b-88cafc087c0b" path="/var/lib/kubelet/pods/5b53b508-7ce8-4320-8f8b-88cafc087c0b/volumes"
Jul 09 04:34:10.484476 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:34:10.484439    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:34:25.591140 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:34:25.591102    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:34:41.380660 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:34:41.380622    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:34:55.851179 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:34:55.851141    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:35:10.089722 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:10.089688    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:35:22.504685 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.504651    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"]
Jul 09 04:35:22.505097 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.505006    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" containerName="main"
Jul 09 04:35:22.505097 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.505017    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" containerName="main"
Jul 09 04:35:22.505097 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.505032    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b53b508-7ce8-4320-8f8b-88cafc087c0b" containerName="main"
Jul 09 04:35:22.505097 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.505039    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b53b508-7ce8-4320-8f8b-88cafc087c0b" containerName="main"
Jul 09 04:35:22.505097 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.505098    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b53b508-7ce8-4320-8f8b-88cafc087c0b" containerName="main"
Jul 09 04:35:22.505286 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.505108    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f715a9-a4e0-46bd-ab31-3c6d565fff0f" containerName="main"
Jul 09 04:35:22.507942 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.507926    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.510446 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.510424    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7b94ae\"/\"kube-root-ca.crt\""
Jul 09 04:35:22.511593 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.511572    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7b94ae\"/\"auth-test-model-7b94ae-kserve-self-signed-certs\""
Jul 09 04:35:22.511709 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.511607    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7b94ae\"/\"openshift-service-ca.crt\""
Jul 09 04:35:22.511709 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.511668    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7b94ae\"/\"default-dockercfg-qf28z\""
Jul 09 04:35:22.517435 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.517414    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"]
Jul 09 04:35:22.639565 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.639530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-home\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.639736 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.639592    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-dshm\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.639736 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.639656    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8937a082-7144-45be-af33-72cf337ae1c9-tls-certs\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.639736 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.639677    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-model-cache\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.639736 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.639695    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w66bd\" (UniqueName: \"kubernetes.io/projected/8937a082-7144-45be-af33-72cf337ae1c9-kube-api-access-w66bd\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.639913 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.639759    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-tmp-dir\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.740869 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.740830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8937a082-7144-45be-af33-72cf337ae1c9-tls-certs\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.740869 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.740868    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-model-cache\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.741117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.740887    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w66bd\" (UniqueName: \"kubernetes.io/projected/8937a082-7144-45be-af33-72cf337ae1c9-kube-api-access-w66bd\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.741117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.740930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-tmp-dir\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.741117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.740966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-home\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.741117 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.741034    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-dshm\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.741369 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.741347    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-model-cache\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.741427 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.741402    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-tmp-dir\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.741427 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.741421    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-home\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.743270 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.743240    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-dshm\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.743531 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.743513    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8937a082-7144-45be-af33-72cf337ae1c9-tls-certs\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.748251 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.748235    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w66bd\" (UniqueName: \"kubernetes.io/projected/8937a082-7144-45be-af33-72cf337ae1c9-kube-api-access-w66bd\") pod \"auth-test-model-7b94ae-kserve-6d8d874bc5-vthng\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") " pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.818263 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.818161    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:22.948647 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:22.948612    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"]
Jul 09 04:35:22.951985 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:35:22.951959    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8937a082_7144_45be_af33_72cf337ae1c9.slice/crio-5c391f3e3e31d938fdc4babac8532b4d88664753d6587bc3a5eb768bf6685b94 WatchSource:0}: Error finding container 5c391f3e3e31d938fdc4babac8532b4d88664753d6587bc3a5eb768bf6685b94: Status 404 returned error can't find the container with id 5c391f3e3e31d938fdc4babac8532b4d88664753d6587bc3a5eb768bf6685b94
Jul 09 04:35:23.675479 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:23.675428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng" event={"ID":"8937a082-7144-45be-af33-72cf337ae1c9","Type":"ContainerStarted","Data":"2621ca3c82573cbe4f26224b5dc159f44dd978f043cec8419cba46fdce7d3a46"}
Jul 09 04:35:23.675895 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:23.675492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng" event={"ID":"8937a082-7144-45be-af33-72cf337ae1c9","Type":"ContainerStarted","Data":"5c391f3e3e31d938fdc4babac8532b4d88664753d6587bc3a5eb768bf6685b94"}
Jul 09 04:35:23.693181 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:23.693134    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng" podStartSLOduration=1.693121266 podStartE2EDuration="1.693121266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:35:23.69216004 +0000 UTC m=+2130.067163871" watchObservedRunningTime="2026-07-09 04:35:23.693121266 +0000 UTC m=+2130.068125084"
Jul 09 04:35:26.284986 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:26.284950    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:35:32.818376 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:32.818337    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:32.818776 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:32.818391    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:32.830832 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:32.830808    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:33.733573 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:33.733543    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:35:37.679357 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.679323    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"]
Jul 09 04:35:37.682667 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.682645    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.686352 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.686333    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"openshift-service-ca.crt\""
Jul 09 04:35:37.686471 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.686369    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"default-dockercfg-zpgxl\""
Jul 09 04:35:37.686471 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.686386    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"auth-test-model-65509e-kserve-self-signed-certs\""
Jul 09 04:35:37.686471 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.686376    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"kube-root-ca.crt\""
Jul 09 04:35:37.692459 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.692439    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"]
Jul 09 04:35:37.759443 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.759413    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-model-cache\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.759616 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.759460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-dshm\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.759616 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.759492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-home\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.759616 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.759541    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tls-certs\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.759616 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.759558    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tmp-dir\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.759616 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.759577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7zf\" (UniqueName: \"kubernetes.io/projected/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-kube-api-access-8n7zf\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.860286 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.860253    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-dshm\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.860468 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.860302    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-home\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.860468 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.860341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tls-certs\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.860468 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.860361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tmp-dir\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.860468 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.860378    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7zf\" (UniqueName: \"kubernetes.io/projected/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-kube-api-access-8n7zf\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.860468 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.860418    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-model-cache\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.860793 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.860765    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-home\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.860883 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.860834    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tmp-dir\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.860883 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.860859    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-model-cache\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.862589 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.862563    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-dshm\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.862836 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.862819    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tls-certs\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.869669 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.869635    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7zf\" (UniqueName: \"kubernetes.io/projected/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-kube-api-access-8n7zf\") pod \"auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") " pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:37.992550 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:37.992464    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:38.124884 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:38.124859    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"]
Jul 09 04:35:38.126194 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:35:38.126167    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb03d5620_fb44_44e9_9e24_6ccb1c4adf75.slice/crio-b0df3c09d10c81fde8013e9f90a7e9253e0f99d37b9cefb717eb885a4cb59f9d WatchSource:0}: Error finding container b0df3c09d10c81fde8013e9f90a7e9253e0f99d37b9cefb717eb885a4cb59f9d: Status 404 returned error can't find the container with id b0df3c09d10c81fde8013e9f90a7e9253e0f99d37b9cefb717eb885a4cb59f9d
Jul 09 04:35:38.746466 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:38.746421    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6" event={"ID":"b03d5620-fb44-44e9-9e24-6ccb1c4adf75","Type":"ContainerStarted","Data":"0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f"}
Jul 09 04:35:38.746836 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:38.746474    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6" event={"ID":"b03d5620-fb44-44e9-9e24-6ccb1c4adf75","Type":"ContainerStarted","Data":"b0df3c09d10c81fde8013e9f90a7e9253e0f99d37b9cefb717eb885a4cb59f9d"}
Jul 09 04:35:38.763012 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:38.762965    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6" podStartSLOduration=1.762950501 podStartE2EDuration="1.762950501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:35:38.761935585 +0000 UTC m=+2145.136939421" watchObservedRunningTime="2026-07-09 04:35:38.762950501 +0000 UTC m=+2145.137954319"
Jul 09 04:35:41.481566 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:41.481529    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:35:47.993141 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:47.993053    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:47.993141 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:47.993115    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:48.005489 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:48.005463    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:35:48.806692 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:35:48.806664    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:36:04.882595 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:36:04.882561    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:36:23.279590 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:36:23.279549    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:36:40.791642 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:36:40.791602    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:36:57.186570 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:36:57.186536    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:37:15.191045 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:37:15.191007    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:37:33.381849 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:37:33.381818    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:37:48.187374 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:37:48.187343    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"]
Jul 09 04:37:49.059763 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:37:49.059731    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"]
Jul 09 04:37:51.481145 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:37:51.481109    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:38:03.207286 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.207241    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng" podUID="8937a082-7144-45be-af33-72cf337ae1c9" containerName="main" containerID="cri-o://2621ca3c82573cbe4f26224b5dc159f44dd978f043cec8419cba46fdce7d3a46" gracePeriod=45
Jul 09 04:38:03.425373 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.425335    2558 generic.go:356] "Generic (PLEG): container finished" podID="8937a082-7144-45be-af33-72cf337ae1c9" containerID="2621ca3c82573cbe4f26224b5dc159f44dd978f043cec8419cba46fdce7d3a46" exitCode=0
Jul 09 04:38:03.425539 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.425399    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng" event={"ID":"8937a082-7144-45be-af33-72cf337ae1c9","Type":"ContainerDied","Data":"2621ca3c82573cbe4f26224b5dc159f44dd978f043cec8419cba46fdce7d3a46"}
Jul 09 04:38:03.461518 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.461456    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:38:03.549521 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.549480    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8937a082-7144-45be-af33-72cf337ae1c9-tls-certs\") pod \"8937a082-7144-45be-af33-72cf337ae1c9\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") "
Jul 09 04:38:03.549521 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.549516    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-model-cache\") pod \"8937a082-7144-45be-af33-72cf337ae1c9\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") "
Jul 09 04:38:03.549766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.549547    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w66bd\" (UniqueName: \"kubernetes.io/projected/8937a082-7144-45be-af33-72cf337ae1c9-kube-api-access-w66bd\") pod \"8937a082-7144-45be-af33-72cf337ae1c9\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") "
Jul 09 04:38:03.549766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.549569    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-tmp-dir\") pod \"8937a082-7144-45be-af33-72cf337ae1c9\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") "
Jul 09 04:38:03.549766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.549603    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-home\") pod \"8937a082-7144-45be-af33-72cf337ae1c9\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") "
Jul 09 04:38:03.549766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.549644    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-dshm\") pod \"8937a082-7144-45be-af33-72cf337ae1c9\" (UID: \"8937a082-7144-45be-af33-72cf337ae1c9\") "
Jul 09 04:38:03.549998 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.549809    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "8937a082-7144-45be-af33-72cf337ae1c9" (UID: "8937a082-7144-45be-af33-72cf337ae1c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:38:03.549998 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.549886    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-home" (OuterVolumeSpecName: "home") pod "8937a082-7144-45be-af33-72cf337ae1c9" (UID: "8937a082-7144-45be-af33-72cf337ae1c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:38:03.549998 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.549892    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8937a082-7144-45be-af33-72cf337ae1c9" (UID: "8937a082-7144-45be-af33-72cf337ae1c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:38:03.551790 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.551766    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-dshm" (OuterVolumeSpecName: "dshm") pod "8937a082-7144-45be-af33-72cf337ae1c9" (UID: "8937a082-7144-45be-af33-72cf337ae1c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:38:03.551902 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.551823    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8937a082-7144-45be-af33-72cf337ae1c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8937a082-7144-45be-af33-72cf337ae1c9" (UID: "8937a082-7144-45be-af33-72cf337ae1c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:38:03.551944 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.551907    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8937a082-7144-45be-af33-72cf337ae1c9-kube-api-access-w66bd" (OuterVolumeSpecName: "kube-api-access-w66bd") pod "8937a082-7144-45be-af33-72cf337ae1c9" (UID: "8937a082-7144-45be-af33-72cf337ae1c9"). InnerVolumeSpecName "kube-api-access-w66bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:38:03.651218 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.651185    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w66bd\" (UniqueName: \"kubernetes.io/projected/8937a082-7144-45be-af33-72cf337ae1c9-kube-api-access-w66bd\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:03.651445 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.651235    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-tmp-dir\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:03.651445 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.651246    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-home\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:03.651445 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.651254    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-dshm\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:03.651445 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.651261    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8937a082-7144-45be-af33-72cf337ae1c9-tls-certs\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:03.651445 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:03.651268    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8937a082-7144-45be-af33-72cf337ae1c9-model-cache\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:04.080744 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.080707    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6" podUID="b03d5620-fb44-44e9-9e24-6ccb1c4adf75" containerName="main" containerID="cri-o://0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f" gracePeriod=45
Jul 09 04:38:04.333813 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.333755    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:38:04.356610 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.356582    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-home\") pod \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") "
Jul 09 04:38:04.356745 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.356616    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n7zf\" (UniqueName: \"kubernetes.io/projected/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-kube-api-access-8n7zf\") pod \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") "
Jul 09 04:38:04.356745 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.356649    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tmp-dir\") pod \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") "
Jul 09 04:38:04.356745 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.356667    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tls-certs\") pod \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") "
Jul 09 04:38:04.356745 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.356733    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-model-cache\") pod \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") "
Jul 09 04:38:04.356931 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.356776    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-dshm\") pod \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\" (UID: \"b03d5620-fb44-44e9-9e24-6ccb1c4adf75\") "
Jul 09 04:38:04.356931 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.356865    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b03d5620-fb44-44e9-9e24-6ccb1c4adf75" (UID: "b03d5620-fb44-44e9-9e24-6ccb1c4adf75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:38:04.356931 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.356900    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-home" (OuterVolumeSpecName: "home") pod "b03d5620-fb44-44e9-9e24-6ccb1c4adf75" (UID: "b03d5620-fb44-44e9-9e24-6ccb1c4adf75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:38:04.357091 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.357040    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-model-cache" (OuterVolumeSpecName: "model-cache") pod "b03d5620-fb44-44e9-9e24-6ccb1c4adf75" (UID: "b03d5620-fb44-44e9-9e24-6ccb1c4adf75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:38:04.357134 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.357112    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-home\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:04.357134 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.357131    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tmp-dir\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:04.357224 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.357145    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-model-cache\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:04.358996 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.358975    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-dshm" (OuterVolumeSpecName: "dshm") pod "b03d5620-fb44-44e9-9e24-6ccb1c4adf75" (UID: "b03d5620-fb44-44e9-9e24-6ccb1c4adf75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:38:04.359355 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.359336    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b03d5620-fb44-44e9-9e24-6ccb1c4adf75" (UID: "b03d5620-fb44-44e9-9e24-6ccb1c4adf75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:38:04.359413 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.359364    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-kube-api-access-8n7zf" (OuterVolumeSpecName: "kube-api-access-8n7zf") pod "b03d5620-fb44-44e9-9e24-6ccb1c4adf75" (UID: "b03d5620-fb44-44e9-9e24-6ccb1c4adf75"). InnerVolumeSpecName "kube-api-access-8n7zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:38:04.431617 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.431587    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"
Jul 09 04:38:04.431789 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.431630    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng" event={"ID":"8937a082-7144-45be-af33-72cf337ae1c9","Type":"ContainerDied","Data":"5c391f3e3e31d938fdc4babac8532b4d88664753d6587bc3a5eb768bf6685b94"}
Jul 09 04:38:04.431789 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.431685    2558 scope.go:117] "RemoveContainer" containerID="2621ca3c82573cbe4f26224b5dc159f44dd978f043cec8419cba46fdce7d3a46"
Jul 09 04:38:04.432942 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.432920    2558 generic.go:356] "Generic (PLEG): container finished" podID="b03d5620-fb44-44e9-9e24-6ccb1c4adf75" containerID="0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f" exitCode=0
Jul 09 04:38:04.433021 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.432953    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"
Jul 09 04:38:04.433021 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.432973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6" event={"ID":"b03d5620-fb44-44e9-9e24-6ccb1c4adf75","Type":"ContainerDied","Data":"0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f"}
Jul 09 04:38:04.433021 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.432995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6" event={"ID":"b03d5620-fb44-44e9-9e24-6ccb1c4adf75","Type":"ContainerDied","Data":"b0df3c09d10c81fde8013e9f90a7e9253e0f99d37b9cefb717eb885a4cb59f9d"}
Jul 09 04:38:04.442372 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.442351    2558 scope.go:117] "RemoveContainer" containerID="0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f"
Jul 09 04:38:04.448875 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.448855    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"]
Jul 09 04:38:04.450829 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.450812    2558 scope.go:117] "RemoveContainer" containerID="0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f"
Jul 09 04:38:04.451083 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:38:04.451064    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f\": container with ID starting with 0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f not found: ID does not exist" containerID="0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f"
Jul 09 04:38:04.451128 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.451093    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f"} err="failed to get container status \"0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f\": rpc error: code = NotFound desc = could not find container \"0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f\": container with ID starting with 0b953d224d2cf7f10418739af4730de1f695f91b02707d91aaf17cba46e3d26f not found: ID does not exist"
Jul 09 04:38:04.454242 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.454222    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/auth-test-model-7b94ae-kserve-6d8d874bc5-vthng"]
Jul 09 04:38:04.457920 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.457903    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-dshm\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:04.458002 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.457924    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n7zf\" (UniqueName: \"kubernetes.io/projected/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-kube-api-access-8n7zf\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:04.458002 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.457935    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03d5620-fb44-44e9-9e24-6ccb1c4adf75-tls-certs\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:38:04.467720 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.467698    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"]
Jul 09 04:38:04.471470 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:04.471451    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/auth-test-model-65509e-kserve-5bbd99b8f9-6zlf6"]
Jul 09 04:38:06.109560 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:06.109526    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8937a082-7144-45be-af33-72cf337ae1c9" path="/var/lib/kubelet/pods/8937a082-7144-45be-af33-72cf337ae1c9/volumes"
Jul 09 04:38:06.109927 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:06.109907    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03d5620-fb44-44e9-9e24-6ccb1c4adf75" path="/var/lib/kubelet/pods/b03d5620-fb44-44e9-9e24-6ccb1c4adf75/volumes"
Jul 09 04:38:16.768812 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.768781    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"]
Jul 09 04:38:16.769378 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.769353    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03d5620-fb44-44e9-9e24-6ccb1c4adf75" containerName="main"
Jul 09 04:38:16.769378 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.769372    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03d5620-fb44-44e9-9e24-6ccb1c4adf75" containerName="main"
Jul 09 04:38:16.769502 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.769388    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8937a082-7144-45be-af33-72cf337ae1c9" containerName="main"
Jul 09 04:38:16.769502 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.769396    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8937a082-7144-45be-af33-72cf337ae1c9" containerName="main"
Jul 09 04:38:16.769502 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.769491    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8937a082-7144-45be-af33-72cf337ae1c9" containerName="main"
Jul 09 04:38:16.769638 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.769506    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03d5620-fb44-44e9-9e24-6ccb1c4adf75" containerName="main"
Jul 09 04:38:16.773958 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.773939    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.776658 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.776630    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7b94ae\"/\"openshift-service-ca.crt\""
Jul 09 04:38:16.776781 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.776694    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7b94ae\"/\"default-dockercfg-qf28z\""
Jul 09 04:38:16.777906 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.777846    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7b94ae\"/\"kube-root-ca.crt\""
Jul 09 04:38:16.778314 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.778292    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7b94ae\"/\"sub-test-model-7b94ae-kserve-self-signed-certs\""
Jul 09 04:38:16.780417 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.780394    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"]
Jul 09 04:38:16.868454 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.868414    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/977c584b-74db-44c3-81eb-5b16a89c55e7-tls-certs\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.868454 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.868455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rw5h\" (UniqueName: \"kubernetes.io/projected/977c584b-74db-44c3-81eb-5b16a89c55e7-kube-api-access-4rw5h\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.868658 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.868512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-model-cache\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.868658 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.868582    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-home\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.868658 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.868630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-tmp-dir\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.868759 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.868658    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-dshm\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.969337 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.969303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-model-cache\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.969487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.969368    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-home\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.969487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.969419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-tmp-dir\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.969487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.969455    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-dshm\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.969622 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.969496    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/977c584b-74db-44c3-81eb-5b16a89c55e7-tls-certs\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.969622 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.969520    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rw5h\" (UniqueName: \"kubernetes.io/projected/977c584b-74db-44c3-81eb-5b16a89c55e7-kube-api-access-4rw5h\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.969811 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.969789    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-model-cache\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.969886 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.969807    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-tmp-dir\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.969886 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.969841    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-home\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.971788 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.971766    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-dshm\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.971972 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.971953    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/977c584b-74db-44c3-81eb-5b16a89c55e7-tls-certs\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:16.979234 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:16.979199    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rw5h\" (UniqueName: \"kubernetes.io/projected/977c584b-74db-44c3-81eb-5b16a89c55e7-kube-api-access-4rw5h\") pod \"sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") " pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:17.084016 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:17.083927    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:17.218606 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:17.218571    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"]
Jul 09 04:38:17.222017 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:38:17.221991    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod977c584b_74db_44c3_81eb_5b16a89c55e7.slice/crio-eb48c1a1402cebc3534c03e21c25aaf8a792b93bb404e41c9f1a517da3c8d5c9 WatchSource:0}: Error finding container eb48c1a1402cebc3534c03e21c25aaf8a792b93bb404e41c9f1a517da3c8d5c9: Status 404 returned error can't find the container with id eb48c1a1402cebc3534c03e21c25aaf8a792b93bb404e41c9f1a517da3c8d5c9
Jul 09 04:38:17.502200 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:17.502160    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7" event={"ID":"977c584b-74db-44c3-81eb-5b16a89c55e7","Type":"ContainerStarted","Data":"bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb"}
Jul 09 04:38:17.502424 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:17.502231    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7" event={"ID":"977c584b-74db-44c3-81eb-5b16a89c55e7","Type":"ContainerStarted","Data":"eb48c1a1402cebc3534c03e21c25aaf8a792b93bb404e41c9f1a517da3c8d5c9"}
Jul 09 04:38:17.522695 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:17.522643    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7" podStartSLOduration=1.52262869 podStartE2EDuration="1.52262869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:38:17.52044665 +0000 UTC m=+2303.895450487" watchObservedRunningTime="2026-07-09 04:38:17.52262869 +0000 UTC m=+2303.897632507"
Jul 09 04:38:20.889833 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:20.889796    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:38:27.085017 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:27.084966    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:27.085442 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:27.085032    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:27.097554 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:27.097533    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:27.560633 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:27.560605    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:38:33.701789 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.701750    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"]
Jul 09 04:38:33.705017 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.704998    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.707766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.707742    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"kube-root-ca.crt\""
Jul 09 04:38:33.708802 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.708782    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"default-dockercfg-zpgxl\""
Jul 09 04:38:33.708912 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.708857    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"openshift-service-ca.crt\""
Jul 09 04:38:33.708912 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.708870    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"sub-test-model-65509e-kserve-self-signed-certs\""
Jul 09 04:38:33.717672 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.717652    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"]
Jul 09 04:38:33.818958 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.818922    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-home\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.818958 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.818958    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-tmp-dir\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.819162 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.818982    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39dad435-f33a-4efa-821d-52c26d33a284-tls-certs\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.819162 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.819009    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvb4b\" (UniqueName: \"kubernetes.io/projected/39dad435-f33a-4efa-821d-52c26d33a284-kube-api-access-vvb4b\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.819162 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.819043    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-dshm\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.819162 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.819098    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-model-cache\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.920485 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.920449    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39dad435-f33a-4efa-821d-52c26d33a284-tls-certs\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.920485 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.920484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvb4b\" (UniqueName: \"kubernetes.io/projected/39dad435-f33a-4efa-821d-52c26d33a284-kube-api-access-vvb4b\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.920713 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.920520    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-dshm\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.920713 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.920579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-model-cache\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.920713 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.920647    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-home\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.920713 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.920670    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-tmp-dir\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.921043 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.921017    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-tmp-dir\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.921121 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.921047    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-home\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.921121 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.921087    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-model-cache\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.922716 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.922696    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-dshm\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.923000 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.922982    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39dad435-f33a-4efa-821d-52c26d33a284-tls-certs\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:33.929320 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:33.929297    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvb4b\" (UniqueName: \"kubernetes.io/projected/39dad435-f33a-4efa-821d-52c26d33a284-kube-api-access-vvb4b\") pod \"sub-test-model-65509e-kserve-5999cd867-fmqdv\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") " pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:34.014801 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:34.014700    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:34.152134 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:34.152107    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"]
Jul 09 04:38:34.154466 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:38:34.154433    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39dad435_f33a_4efa_821d_52c26d33a284.slice/crio-0ca92777d56aeaf5c43b8c93df1b1726df7c184c3346462e8ba3e92337db47ad WatchSource:0}: Error finding container 0ca92777d56aeaf5c43b8c93df1b1726df7c184c3346462e8ba3e92337db47ad: Status 404 returned error can't find the container with id 0ca92777d56aeaf5c43b8c93df1b1726df7c184c3346462e8ba3e92337db47ad
Jul 09 04:38:34.582291 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:34.582257    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv" event={"ID":"39dad435-f33a-4efa-821d-52c26d33a284","Type":"ContainerStarted","Data":"90169b56300a2b83dd50c2c967ff6c7a10cd5c2af4eaf551a518a0ed6ab4d48f"}
Jul 09 04:38:34.582478 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:34.582303    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv" event={"ID":"39dad435-f33a-4efa-821d-52c26d33a284","Type":"ContainerStarted","Data":"0ca92777d56aeaf5c43b8c93df1b1726df7c184c3346462e8ba3e92337db47ad"}
Jul 09 04:38:34.600505 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:34.600452    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv" podStartSLOduration=1.600436925 podStartE2EDuration="1.600436925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:38:34.599496798 +0000 UTC m=+2320.974500638" watchObservedRunningTime="2026-07-09 04:38:34.600436925 +0000 UTC m=+2320.975440758"
Jul 09 04:38:39.198410 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:39.198371    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:38:44.015800 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:44.015747    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:44.016327 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:44.015820    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:44.028457 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:44.028431    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:44.637696 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:44.637667    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:38:54.297822 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:38:54.297796    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:39:10.081153 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:10.081121    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:39:14.714967 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:14.714933    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"]
Jul 09 04:39:15.692277 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:15.692246    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"]
Jul 09 04:39:25.972790 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:25.972757    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"]
Jul 09 04:39:25.976287 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:25.976264    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:25.979300 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:25.979280    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7b94ae\"/\"rate-test-model-7b94ae-kserve-self-signed-certs\""
Jul 09 04:39:25.985793 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:25.985773    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"]
Jul 09 04:39:26.078670 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.078630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-dshm\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.078670 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.078668    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-tmp-dir\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.078896 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.078689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061b0065-75cd-4ad8-95cf-4a709f345c14-tls-certs\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.078896 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.078841    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfdb\" (UniqueName: \"kubernetes.io/projected/061b0065-75cd-4ad8-95cf-4a709f345c14-kube-api-access-rqfdb\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.078896 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.078878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-model-cache\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.079006 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.078902    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-home\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.179560 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.179516    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-dshm\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.179766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.179569    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-tmp-dir\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.179766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.179596    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061b0065-75cd-4ad8-95cf-4a709f345c14-tls-certs\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.179766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.179722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfdb\" (UniqueName: \"kubernetes.io/projected/061b0065-75cd-4ad8-95cf-4a709f345c14-kube-api-access-rqfdb\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.179766 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.179755    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-model-cache\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.179995 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.179789    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-home\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.179995 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.179949    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-tmp-dir\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.180104 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.180081    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-home\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.180156 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.180138    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-model-cache\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.182003 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.181976    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-dshm\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.182151 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.182134    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061b0065-75cd-4ad8-95cf-4a709f345c14-tls-certs\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.187673 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.187651    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfdb\" (UniqueName: \"kubernetes.io/projected/061b0065-75cd-4ad8-95cf-4a709f345c14-kube-api-access-rqfdb\") pod \"rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") " pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.285635 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.285546    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:26.418695 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.418672    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"]
Jul 09 04:39:26.420991 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:39:26.420962    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod061b0065_75cd_4ad8_95cf_4a709f345c14.slice/crio-ff9e1c8d977032534c12f14d0a4be904ae499501ce039d18395f66dae09dcb80 WatchSource:0}: Error finding container ff9e1c8d977032534c12f14d0a4be904ae499501ce039d18395f66dae09dcb80: Status 404 returned error can't find the container with id ff9e1c8d977032534c12f14d0a4be904ae499501ce039d18395f66dae09dcb80
Jul 09 04:39:26.818994 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.818963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d" event={"ID":"061b0065-75cd-4ad8-95cf-4a709f345c14","Type":"ContainerStarted","Data":"c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3"}
Jul 09 04:39:26.819338 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.819004    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d" event={"ID":"061b0065-75cd-4ad8-95cf-4a709f345c14","Type":"ContainerStarted","Data":"ff9e1c8d977032534c12f14d0a4be904ae499501ce039d18395f66dae09dcb80"}
Jul 09 04:39:26.837193 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:26.837142    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d" podStartSLOduration=1.837126605 podStartE2EDuration="1.837126605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:39:26.836406039 +0000 UTC m=+2373.211409895" watchObservedRunningTime="2026-07-09 04:39:26.837126605 +0000 UTC m=+2373.212130422"
Jul 09 04:39:28.785497 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:28.785460    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:39:29.734787 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:29.734741    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7" podUID="977c584b-74db-44c3-81eb-5b16a89c55e7" containerName="main" containerID="cri-o://bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb" gracePeriod=45
Jul 09 04:39:29.988246 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:29.988187    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:39:30.118856 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.118822    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-tmp-dir\") pod \"977c584b-74db-44c3-81eb-5b16a89c55e7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") "
Jul 09 04:39:30.118856 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.118861    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-home\") pod \"977c584b-74db-44c3-81eb-5b16a89c55e7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") "
Jul 09 04:39:30.119088 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.118935    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-model-cache\") pod \"977c584b-74db-44c3-81eb-5b16a89c55e7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") "
Jul 09 04:39:30.119088 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.118972    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/977c584b-74db-44c3-81eb-5b16a89c55e7-tls-certs\") pod \"977c584b-74db-44c3-81eb-5b16a89c55e7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") "
Jul 09 04:39:30.119088 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.119017    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-dshm\") pod \"977c584b-74db-44c3-81eb-5b16a89c55e7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") "
Jul 09 04:39:30.119268 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.119104    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "977c584b-74db-44c3-81eb-5b16a89c55e7" (UID: "977c584b-74db-44c3-81eb-5b16a89c55e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:39:30.119268 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.119192    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "977c584b-74db-44c3-81eb-5b16a89c55e7" (UID: "977c584b-74db-44c3-81eb-5b16a89c55e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:39:30.119268 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.119201    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-home" (OuterVolumeSpecName: "home") pod "977c584b-74db-44c3-81eb-5b16a89c55e7" (UID: "977c584b-74db-44c3-81eb-5b16a89c55e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:39:30.119424 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.119409    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rw5h\" (UniqueName: \"kubernetes.io/projected/977c584b-74db-44c3-81eb-5b16a89c55e7-kube-api-access-4rw5h\") pod \"977c584b-74db-44c3-81eb-5b16a89c55e7\" (UID: \"977c584b-74db-44c3-81eb-5b16a89c55e7\") "
Jul 09 04:39:30.119770 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.119714    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-model-cache\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:30.119770 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.119737    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-tmp-dir\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:30.119770 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.119751    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-home\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:30.121076 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.121053    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-dshm" (OuterVolumeSpecName: "dshm") pod "977c584b-74db-44c3-81eb-5b16a89c55e7" (UID: "977c584b-74db-44c3-81eb-5b16a89c55e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:39:30.121331 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.121309    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/977c584b-74db-44c3-81eb-5b16a89c55e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "977c584b-74db-44c3-81eb-5b16a89c55e7" (UID: "977c584b-74db-44c3-81eb-5b16a89c55e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:39:30.121422 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.121311    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977c584b-74db-44c3-81eb-5b16a89c55e7-kube-api-access-4rw5h" (OuterVolumeSpecName: "kube-api-access-4rw5h") pod "977c584b-74db-44c3-81eb-5b16a89c55e7" (UID: "977c584b-74db-44c3-81eb-5b16a89c55e7"). InnerVolumeSpecName "kube-api-access-4rw5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:39:30.220436 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.220401    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/977c584b-74db-44c3-81eb-5b16a89c55e7-tls-certs\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:30.220436 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.220430    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/977c584b-74db-44c3-81eb-5b16a89c55e7-dshm\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:30.220436 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.220440    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rw5h\" (UniqueName: \"kubernetes.io/projected/977c584b-74db-44c3-81eb-5b16a89c55e7-kube-api-access-4rw5h\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:30.712322 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.712274    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv" podUID="39dad435-f33a-4efa-821d-52c26d33a284" containerName="main" containerID="cri-o://90169b56300a2b83dd50c2c967ff6c7a10cd5c2af4eaf551a518a0ed6ab4d48f" gracePeriod=45
Jul 09 04:39:30.842438 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.842401    2558 generic.go:356] "Generic (PLEG): container finished" podID="39dad435-f33a-4efa-821d-52c26d33a284" containerID="90169b56300a2b83dd50c2c967ff6c7a10cd5c2af4eaf551a518a0ed6ab4d48f" exitCode=0
Jul 09 04:39:30.842600 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.842469    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv" event={"ID":"39dad435-f33a-4efa-821d-52c26d33a284","Type":"ContainerDied","Data":"90169b56300a2b83dd50c2c967ff6c7a10cd5c2af4eaf551a518a0ed6ab4d48f"}
Jul 09 04:39:30.843721 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.843696    2558 generic.go:356] "Generic (PLEG): container finished" podID="977c584b-74db-44c3-81eb-5b16a89c55e7" containerID="bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb" exitCode=0
Jul 09 04:39:30.843852 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.843743    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"
Jul 09 04:39:30.843852 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.843745    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7" event={"ID":"977c584b-74db-44c3-81eb-5b16a89c55e7","Type":"ContainerDied","Data":"bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb"}
Jul 09 04:39:30.843852 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.843778    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7" event={"ID":"977c584b-74db-44c3-81eb-5b16a89c55e7","Type":"ContainerDied","Data":"eb48c1a1402cebc3534c03e21c25aaf8a792b93bb404e41c9f1a517da3c8d5c9"}
Jul 09 04:39:30.843852 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.843794    2558 scope.go:117] "RemoveContainer" containerID="bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb"
Jul 09 04:39:30.852961 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.852937    2558 scope.go:117] "RemoveContainer" containerID="bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb"
Jul 09 04:39:30.853269 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:39:30.853248    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb\": container with ID starting with bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb not found: ID does not exist" containerID="bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb"
Jul 09 04:39:30.853347 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.853278    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb"} err="failed to get container status \"bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb\": rpc error: code = NotFound desc = could not find container \"bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb\": container with ID starting with bb21a2ec406ce36eb88a2f4c000b23dd631ad7b3a65f4e1ba4405e9b024e48cb not found: ID does not exist"
Jul 09 04:39:30.865096 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.865071    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"]
Jul 09 04:39:30.868997 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.868974    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/sub-test-model-7b94ae-kserve-7f7f8c58dc-j9qn7"]
Jul 09 04:39:30.964457 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:30.964403    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:39:31.026199 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.026168    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-home\") pod \"39dad435-f33a-4efa-821d-52c26d33a284\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") "
Jul 09 04:39:31.026199 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.026202    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39dad435-f33a-4efa-821d-52c26d33a284-tls-certs\") pod \"39dad435-f33a-4efa-821d-52c26d33a284\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") "
Jul 09 04:39:31.026715 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.026248    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-tmp-dir\") pod \"39dad435-f33a-4efa-821d-52c26d33a284\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") "
Jul 09 04:39:31.026715 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.026264    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-dshm\") pod \"39dad435-f33a-4efa-821d-52c26d33a284\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") "
Jul 09 04:39:31.026715 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.026292    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvb4b\" (UniqueName: \"kubernetes.io/projected/39dad435-f33a-4efa-821d-52c26d33a284-kube-api-access-vvb4b\") pod \"39dad435-f33a-4efa-821d-52c26d33a284\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") "
Jul 09 04:39:31.026715 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.026485    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-home" (OuterVolumeSpecName: "home") pod "39dad435-f33a-4efa-821d-52c26d33a284" (UID: "39dad435-f33a-4efa-821d-52c26d33a284"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:39:31.026715 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.026552    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39dad435-f33a-4efa-821d-52c26d33a284" (UID: "39dad435-f33a-4efa-821d-52c26d33a284"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:39:31.028437 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.028416    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-dshm" (OuterVolumeSpecName: "dshm") pod "39dad435-f33a-4efa-821d-52c26d33a284" (UID: "39dad435-f33a-4efa-821d-52c26d33a284"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:39:31.028523 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.028473    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dad435-f33a-4efa-821d-52c26d33a284-kube-api-access-vvb4b" (OuterVolumeSpecName: "kube-api-access-vvb4b") pod "39dad435-f33a-4efa-821d-52c26d33a284" (UID: "39dad435-f33a-4efa-821d-52c26d33a284"). InnerVolumeSpecName "kube-api-access-vvb4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:39:31.028560 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.028535    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39dad435-f33a-4efa-821d-52c26d33a284-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39dad435-f33a-4efa-821d-52c26d33a284" (UID: "39dad435-f33a-4efa-821d-52c26d33a284"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:39:31.127178 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.127131    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-model-cache\") pod \"39dad435-f33a-4efa-821d-52c26d33a284\" (UID: \"39dad435-f33a-4efa-821d-52c26d33a284\") "
Jul 09 04:39:31.127412 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.127399    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-home\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:31.127456 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.127415    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39dad435-f33a-4efa-821d-52c26d33a284-tls-certs\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:31.127456 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.127424    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-tmp-dir\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:31.127456 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.127436    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-dshm\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:31.127456 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.127445    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvb4b\" (UniqueName: \"kubernetes.io/projected/39dad435-f33a-4efa-821d-52c26d33a284-kube-api-access-vvb4b\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:31.127456 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.127415    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-model-cache" (OuterVolumeSpecName: "model-cache") pod "39dad435-f33a-4efa-821d-52c26d33a284" (UID: "39dad435-f33a-4efa-821d-52c26d33a284"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:39:31.228022 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.227936    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39dad435-f33a-4efa-821d-52c26d33a284-model-cache\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:39:31.849098 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.849063    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"
Jul 09 04:39:31.849486 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.849068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv" event={"ID":"39dad435-f33a-4efa-821d-52c26d33a284","Type":"ContainerDied","Data":"0ca92777d56aeaf5c43b8c93df1b1726df7c184c3346462e8ba3e92337db47ad"}
Jul 09 04:39:31.849486 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.849216    2558 scope.go:117] "RemoveContainer" containerID="90169b56300a2b83dd50c2c967ff6c7a10cd5c2af4eaf551a518a0ed6ab4d48f"
Jul 09 04:39:31.869596 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.869570    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"]
Jul 09 04:39:31.873107 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:31.873087    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/sub-test-model-65509e-kserve-5999cd867-fmqdv"]
Jul 09 04:39:32.110005 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:32.109918    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39dad435-f33a-4efa-821d-52c26d33a284" path="/var/lib/kubelet/pods/39dad435-f33a-4efa-821d-52c26d33a284/volumes"
Jul 09 04:39:32.110401 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:32.110359    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977c584b-74db-44c3-81eb-5b16a89c55e7" path="/var/lib/kubelet/pods/977c584b-74db-44c3-81eb-5b16a89c55e7/volumes"
Jul 09 04:39:36.286197 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:36.286154    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:36.286619 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:36.286231    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:36.298930 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:36.298903    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:39:36.885439 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:39:36.885409    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:40:17.768440 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.768364    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"]
Jul 09 04:40:17.768859 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.768743    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977c584b-74db-44c3-81eb-5b16a89c55e7" containerName="main"
Jul 09 04:40:17.768859 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.768758    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="977c584b-74db-44c3-81eb-5b16a89c55e7" containerName="main"
Jul 09 04:40:17.768859 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.768771    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39dad435-f33a-4efa-821d-52c26d33a284" containerName="main"
Jul 09 04:40:17.768859 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.768776    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="39dad435-f33a-4efa-821d-52c26d33a284" containerName="main"
Jul 09 04:40:17.768859 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.768841    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="977c584b-74db-44c3-81eb-5b16a89c55e7" containerName="main"
Jul 09 04:40:17.768859 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.768851    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="39dad435-f33a-4efa-821d-52c26d33a284" containerName="main"
Jul 09 04:40:17.776994 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.776955    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:17.781066 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.781040    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"kube-root-ca.crt\""
Jul 09 04:40:17.781431 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.781040    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"openshift-service-ca.crt\""
Jul 09 04:40:17.781554 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.781107    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"rate-test-model-65509e-kserve-self-signed-certs\""
Jul 09 04:40:17.781637 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.781154    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-65509e\"/\"default-dockercfg-zpgxl\""
Jul 09 04:40:17.783059 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.783034    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"]
Jul 09 04:40:17.916772 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.916729    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvn2s\" (UniqueName: \"kubernetes.io/projected/c389f50e-6e72-4ab3-8b85-beec9a5d0785-kube-api-access-lvn2s\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:17.916772 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.916772    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-home\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:17.917003 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.916836    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tls-certs\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:17.917003 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.916875    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-model-cache\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:17.917003 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.916903    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tmp-dir\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:17.917003 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:17.916933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-dshm\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.018300 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.018262    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvn2s\" (UniqueName: \"kubernetes.io/projected/c389f50e-6e72-4ab3-8b85-beec9a5d0785-kube-api-access-lvn2s\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.018300 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.018303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-home\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.018590 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.018338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tls-certs\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.018590 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.018363    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-model-cache\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.018590 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.018388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tmp-dir\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.018590 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.018419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-dshm\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.018802 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.018704    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-home\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.018802 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.018719    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-model-cache\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.018802 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.018793    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tmp-dir\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.020679 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.020661    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-dshm\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.020867 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.020850    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tls-certs\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.027313 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.027283    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvn2s\" (UniqueName: \"kubernetes.io/projected/c389f50e-6e72-4ab3-8b85-beec9a5d0785-kube-api-access-lvn2s\") pod \"rate-test-model-65509e-kserve-54c5674b7-jdm2h\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") " pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.089419 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.089388    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:18.224236 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:18.224196    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"]
Jul 09 04:40:18.226149 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:40:18.226113    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc389f50e_6e72_4ab3_8b85_beec9a5d0785.slice/crio-3d4d84752a46f76475f29aaa8babc11230d0251aa47831e87a3703c68595574f WatchSource:0}: Error finding container 3d4d84752a46f76475f29aaa8babc11230d0251aa47831e87a3703c68595574f: Status 404 returned error can't find the container with id 3d4d84752a46f76475f29aaa8babc11230d0251aa47831e87a3703c68595574f
Jul 09 04:40:19.073422 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:19.073391    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h" event={"ID":"c389f50e-6e72-4ab3-8b85-beec9a5d0785","Type":"ContainerStarted","Data":"ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b"}
Jul 09 04:40:19.073422 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:19.073435    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h" event={"ID":"c389f50e-6e72-4ab3-8b85-beec9a5d0785","Type":"ContainerStarted","Data":"3d4d84752a46f76475f29aaa8babc11230d0251aa47831e87a3703c68595574f"}
Jul 09 04:40:19.090576 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:19.090523    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h" podStartSLOduration=2.090506981 podStartE2EDuration="2.090506981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 04:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:40:19.089631998 +0000 UTC m=+2425.464635830" watchObservedRunningTime="2026-07-09 04:40:19.090506981 +0000 UTC m=+2425.465510798"
Jul 09 04:40:22.887449 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:22.887413    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:40:28.089845 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:28.089802    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:28.090366 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:28.089873    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:28.102469 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:28.102439    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:28.128782 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:28.128752    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:40:48.587400 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:40:48.587366    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:41:11.078516 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:11.078479    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:41:15.367715 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:15.367679    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"]
Jul 09 04:41:16.388633 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:16.388594    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"]
Jul 09 04:41:27.679853 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:27.679813    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:41:30.389656 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.389593    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d" podUID="061b0065-75cd-4ad8-95cf-4a709f345c14" containerName="main" containerID="cri-o://c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3" gracePeriod=45
Jul 09 04:41:30.643960 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.643937    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:41:30.668827 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.668799    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-model-cache\") pod \"061b0065-75cd-4ad8-95cf-4a709f345c14\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") "
Jul 09 04:41:30.668981 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.668875    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061b0065-75cd-4ad8-95cf-4a709f345c14-tls-certs\") pod \"061b0065-75cd-4ad8-95cf-4a709f345c14\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") "
Jul 09 04:41:30.668981 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.668927    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-dshm\") pod \"061b0065-75cd-4ad8-95cf-4a709f345c14\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") "
Jul 09 04:41:30.668981 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.668961    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-tmp-dir\") pod \"061b0065-75cd-4ad8-95cf-4a709f345c14\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") "
Jul 09 04:41:30.669145 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.669012    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqfdb\" (UniqueName: \"kubernetes.io/projected/061b0065-75cd-4ad8-95cf-4a709f345c14-kube-api-access-rqfdb\") pod \"061b0065-75cd-4ad8-95cf-4a709f345c14\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") "
Jul 09 04:41:30.669145 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.669035    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-home\") pod \"061b0065-75cd-4ad8-95cf-4a709f345c14\" (UID: \"061b0065-75cd-4ad8-95cf-4a709f345c14\") "
Jul 09 04:41:30.669145 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.669046    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-model-cache" (OuterVolumeSpecName: "model-cache") pod "061b0065-75cd-4ad8-95cf-4a709f345c14" (UID: "061b0065-75cd-4ad8-95cf-4a709f345c14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:41:30.669321 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.669287    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "061b0065-75cd-4ad8-95cf-4a709f345c14" (UID: "061b0065-75cd-4ad8-95cf-4a709f345c14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:41:30.669321 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.669299    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-model-cache\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:30.669505 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.669481    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-home" (OuterVolumeSpecName: "home") pod "061b0065-75cd-4ad8-95cf-4a709f345c14" (UID: "061b0065-75cd-4ad8-95cf-4a709f345c14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:41:30.671130 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.671107    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061b0065-75cd-4ad8-95cf-4a709f345c14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "061b0065-75cd-4ad8-95cf-4a709f345c14" (UID: "061b0065-75cd-4ad8-95cf-4a709f345c14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:41:30.671262 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.671190    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-dshm" (OuterVolumeSpecName: "dshm") pod "061b0065-75cd-4ad8-95cf-4a709f345c14" (UID: "061b0065-75cd-4ad8-95cf-4a709f345c14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:41:30.671262 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.671221    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061b0065-75cd-4ad8-95cf-4a709f345c14-kube-api-access-rqfdb" (OuterVolumeSpecName: "kube-api-access-rqfdb") pod "061b0065-75cd-4ad8-95cf-4a709f345c14" (UID: "061b0065-75cd-4ad8-95cf-4a709f345c14"). InnerVolumeSpecName "kube-api-access-rqfdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:41:30.770553 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.770518    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-home\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:30.770553 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.770547    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/061b0065-75cd-4ad8-95cf-4a709f345c14-tls-certs\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:30.770553 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.770556    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-dshm\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:30.770553 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.770565    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/061b0065-75cd-4ad8-95cf-4a709f345c14-tmp-dir\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:30.770824 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:30.770574    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfdb\" (UniqueName: \"kubernetes.io/projected/061b0065-75cd-4ad8-95cf-4a709f345c14-kube-api-access-rqfdb\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:31.407826 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.407790    2558 generic.go:356] "Generic (PLEG): container finished" podID="061b0065-75cd-4ad8-95cf-4a709f345c14" containerID="c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3" exitCode=0
Jul 09 04:41:31.408318 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.407848    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"
Jul 09 04:41:31.408318 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.407882    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d" event={"ID":"061b0065-75cd-4ad8-95cf-4a709f345c14","Type":"ContainerDied","Data":"c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3"}
Jul 09 04:41:31.408318 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.407931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d" event={"ID":"061b0065-75cd-4ad8-95cf-4a709f345c14","Type":"ContainerDied","Data":"ff9e1c8d977032534c12f14d0a4be904ae499501ce039d18395f66dae09dcb80"}
Jul 09 04:41:31.408318 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.407950    2558 scope.go:117] "RemoveContainer" containerID="c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3"
Jul 09 04:41:31.414048 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.414013    2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h" podUID="c389f50e-6e72-4ab3-8b85-beec9a5d0785" containerName="main" containerID="cri-o://ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b" gracePeriod=45
Jul 09 04:41:31.418624 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.418521    2558 scope.go:117] "RemoveContainer" containerID="c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3"
Jul 09 04:41:31.418835 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:41:31.418808    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3\": container with ID starting with c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3 not found: ID does not exist" containerID="c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3"
Jul 09 04:41:31.418935 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.418847    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3"} err="failed to get container status \"c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3\": rpc error: code = NotFound desc = could not find container \"c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3\": container with ID starting with c0d02604854755253e9e39c4d77b2e152be1c29ca1aa0f97d815de725fb1a2c3 not found: ID does not exist"
Jul 09 04:41:31.435235 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.434812    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"]
Jul 09 04:41:31.438733 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.438711    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7b94ae/rate-test-model-7b94ae-kserve-5fd968cb75-6mt8d"]
Jul 09 04:41:31.452129 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:41:31.452096    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc389f50e_6e72_4ab3_8b85_beec9a5d0785.slice/crio-ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod061b0065_75cd_4ad8_95cf_4a709f345c14.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod061b0065_75cd_4ad8_95cf_4a709f345c14.slice/crio-ff9e1c8d977032534c12f14d0a4be904ae499501ce039d18395f66dae09dcb80\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc389f50e_6e72_4ab3_8b85_beec9a5d0785.slice/crio-conmon-ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b.scope\": RecentStats: unable to find data in memory cache]"
Jul 09 04:41:31.669540 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.669482    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:41:31.779381 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.779346    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvn2s\" (UniqueName: \"kubernetes.io/projected/c389f50e-6e72-4ab3-8b85-beec9a5d0785-kube-api-access-lvn2s\") pod \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") "
Jul 09 04:41:31.779585 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.779410    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-model-cache\") pod \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") "
Jul 09 04:41:31.779585 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.779430    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-dshm\") pod \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") "
Jul 09 04:41:31.779585 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.779470    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tls-certs\") pod \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") "
Jul 09 04:41:31.779585 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.779517    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-home\") pod \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") "
Jul 09 04:41:31.779585 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.779543    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tmp-dir\") pod \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\" (UID: \"c389f50e-6e72-4ab3-8b85-beec9a5d0785\") "
Jul 09 04:41:31.779848 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.779705    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-model-cache" (OuterVolumeSpecName: "model-cache") pod "c389f50e-6e72-4ab3-8b85-beec9a5d0785" (UID: "c389f50e-6e72-4ab3-8b85-beec9a5d0785"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:41:31.779848 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.779816    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-home" (OuterVolumeSpecName: "home") pod "c389f50e-6e72-4ab3-8b85-beec9a5d0785" (UID: "c389f50e-6e72-4ab3-8b85-beec9a5d0785"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:41:31.779921 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.779848    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c389f50e-6e72-4ab3-8b85-beec9a5d0785" (UID: "c389f50e-6e72-4ab3-8b85-beec9a5d0785"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:41:31.781609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.781586    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-dshm" (OuterVolumeSpecName: "dshm") pod "c389f50e-6e72-4ab3-8b85-beec9a5d0785" (UID: "c389f50e-6e72-4ab3-8b85-beec9a5d0785"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 04:41:31.781711 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.781662    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c389f50e-6e72-4ab3-8b85-beec9a5d0785" (UID: "c389f50e-6e72-4ab3-8b85-beec9a5d0785"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 04:41:31.781768 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.781735    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c389f50e-6e72-4ab3-8b85-beec9a5d0785-kube-api-access-lvn2s" (OuterVolumeSpecName: "kube-api-access-lvn2s") pod "c389f50e-6e72-4ab3-8b85-beec9a5d0785" (UID: "c389f50e-6e72-4ab3-8b85-beec9a5d0785"). InnerVolumeSpecName "kube-api-access-lvn2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:41:31.880398 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.880360    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tls-certs\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:31.880398 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.880392    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-home\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:31.880609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.880418    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-tmp-dir\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:31.880609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.880428    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvn2s\" (UniqueName: \"kubernetes.io/projected/c389f50e-6e72-4ab3-8b85-beec9a5d0785-kube-api-access-lvn2s\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:31.880609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.880439    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-model-cache\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:31.880609 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:31.880448    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c389f50e-6e72-4ab3-8b85-beec9a5d0785-dshm\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:32.109662 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.109579    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="061b0065-75cd-4ad8-95cf-4a709f345c14" path="/var/lib/kubelet/pods/061b0065-75cd-4ad8-95cf-4a709f345c14/volumes"
Jul 09 04:41:32.413443 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.413411    2558 generic.go:356] "Generic (PLEG): container finished" podID="c389f50e-6e72-4ab3-8b85-beec9a5d0785" containerID="ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b" exitCode=0
Jul 09 04:41:32.413952 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.413453    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"
Jul 09 04:41:32.413952 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.413492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h" event={"ID":"c389f50e-6e72-4ab3-8b85-beec9a5d0785","Type":"ContainerDied","Data":"ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b"}
Jul 09 04:41:32.413952 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.413533    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h" event={"ID":"c389f50e-6e72-4ab3-8b85-beec9a5d0785","Type":"ContainerDied","Data":"3d4d84752a46f76475f29aaa8babc11230d0251aa47831e87a3703c68595574f"}
Jul 09 04:41:32.413952 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.413554    2558 scope.go:117] "RemoveContainer" containerID="ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b"
Jul 09 04:41:32.421962 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.421945    2558 scope.go:117] "RemoveContainer" containerID="ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b"
Jul 09 04:41:32.422258 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:41:32.422240    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b\": container with ID starting with ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b not found: ID does not exist" containerID="ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b"
Jul 09 04:41:32.422312 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.422266    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b"} err="failed to get container status \"ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b\": rpc error: code = NotFound desc = could not find container \"ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b\": container with ID starting with ac457e9358fa7f037755fd87eed4d221b628f03c2fbd626b2719cf5af5027c6b not found: ID does not exist"
Jul 09 04:41:32.428856 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.428828    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"]
Jul 09 04:41:32.430723 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:32.430704    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-65509e/rate-test-model-65509e-kserve-54c5674b7-jdm2h"]
Jul 09 04:41:33.294604 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.294570    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7530-140361441258352"]
Jul 09 04:41:33.294985 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.294967    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061b0065-75cd-4ad8-95cf-4a709f345c14" containerName="main"
Jul 09 04:41:33.295078 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.294988    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="061b0065-75cd-4ad8-95cf-4a709f345c14" containerName="main"
Jul 09 04:41:33.295078 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.295004    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c389f50e-6e72-4ab3-8b85-beec9a5d0785" containerName="main"
Jul 09 04:41:33.295078 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.295012    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c389f50e-6e72-4ab3-8b85-beec9a5d0785" containerName="main"
Jul 09 04:41:33.295254 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.295119    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c389f50e-6e72-4ab3-8b85-beec9a5d0785" containerName="main"
Jul 09 04:41:33.295254 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.295138    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="061b0065-75cd-4ad8-95cf-4a709f345c14" containerName="main"
Jul 09 04:41:33.299556 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.299534    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7530-140361441258352"
Jul 09 04:41:33.302300 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.302276    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-d5mg6\""
Jul 09 04:41:33.302854 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.302836    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7530-140361441258352"]
Jul 09 04:41:33.393741 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.393712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9vw\" (UniqueName: \"kubernetes.io/projected/c0422ce4-af95-4350-b3d1-bb20788bfcbc-kube-api-access-qh9vw\") pod \"test-curl-7530-140361441258352\" (UID: \"c0422ce4-af95-4350-b3d1-bb20788bfcbc\") " pod="openshift-ingress/test-curl-7530-140361441258352"
Jul 09 04:41:33.494312 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.494275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9vw\" (UniqueName: \"kubernetes.io/projected/c0422ce4-af95-4350-b3d1-bb20788bfcbc-kube-api-access-qh9vw\") pod \"test-curl-7530-140361441258352\" (UID: \"c0422ce4-af95-4350-b3d1-bb20788bfcbc\") " pod="openshift-ingress/test-curl-7530-140361441258352"
Jul 09 04:41:33.503116 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.503089    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9vw\" (UniqueName: \"kubernetes.io/projected/c0422ce4-af95-4350-b3d1-bb20788bfcbc-kube-api-access-qh9vw\") pod \"test-curl-7530-140361441258352\" (UID: \"c0422ce4-af95-4350-b3d1-bb20788bfcbc\") " pod="openshift-ingress/test-curl-7530-140361441258352"
Jul 09 04:41:33.610574 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.610480    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7530-140361441258352"
Jul 09 04:41:33.741139 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.741103    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7530-140361441258352"]
Jul 09 04:41:33.743868 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:41:33.743843    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0422ce4_af95_4350_b3d1_bb20788bfcbc.slice/crio-ed25118818b16ec07677f734af92803e66a41a24390df7c39524e2460a91778f WatchSource:0}: Error finding container ed25118818b16ec07677f734af92803e66a41a24390df7c39524e2460a91778f: Status 404 returned error can't find the container with id ed25118818b16ec07677f734af92803e66a41a24390df7c39524e2460a91778f
Jul 09 04:41:33.745487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:33.745470    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 04:41:34.112356 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:34.112320    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c389f50e-6e72-4ab3-8b85-beec9a5d0785" path="/var/lib/kubelet/pods/c389f50e-6e72-4ab3-8b85-beec9a5d0785/volumes"
Jul 09 04:41:34.425032 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:34.425000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7530-140361441258352" event={"ID":"c0422ce4-af95-4350-b3d1-bb20788bfcbc","Type":"ContainerStarted","Data":"ed25118818b16ec07677f734af92803e66a41a24390df7c39524e2460a91778f"}
Jul 09 04:41:35.431264 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:35.431228    2558 generic.go:356] "Generic (PLEG): container finished" podID="c0422ce4-af95-4350-b3d1-bb20788bfcbc" containerID="23fd1ed846cf44e99bdf2a1acb36c97325df403bfcec25061c80b8ad3183b48b" exitCode=0
Jul 09 04:41:35.431671 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:35.431312    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7530-140361441258352" event={"ID":"c0422ce4-af95-4350-b3d1-bb20788bfcbc","Type":"ContainerDied","Data":"23fd1ed846cf44e99bdf2a1acb36c97325df403bfcec25061c80b8ad3183b48b"}
Jul 09 04:41:36.572912 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:36.572881    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7530-140361441258352"
Jul 09 04:41:36.606514 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:36.606479    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7530-140361441258352_c0422ce4-af95-4350-b3d1-bb20788bfcbc/test-curl-7530-140361441258352/0.log"
Jul 09 04:41:36.624000 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:36.623969    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh9vw\" (UniqueName: \"kubernetes.io/projected/c0422ce4-af95-4350-b3d1-bb20788bfcbc-kube-api-access-qh9vw\") pod \"c0422ce4-af95-4350-b3d1-bb20788bfcbc\" (UID: \"c0422ce4-af95-4350-b3d1-bb20788bfcbc\") "
Jul 09 04:41:36.626982 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:36.626942    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7530-140361441258352"]
Jul 09 04:41:36.627129 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:36.627078    2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0422ce4-af95-4350-b3d1-bb20788bfcbc-kube-api-access-qh9vw" (OuterVolumeSpecName: "kube-api-access-qh9vw") pod "c0422ce4-af95-4350-b3d1-bb20788bfcbc" (UID: "c0422ce4-af95-4350-b3d1-bb20788bfcbc"). InnerVolumeSpecName "kube-api-access-qh9vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 04:41:36.629034 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:36.629008    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7530-140361441258352"]
Jul 09 04:41:36.725454 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:36.725373    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9vw\" (UniqueName: \"kubernetes.io/projected/c0422ce4-af95-4350-b3d1-bb20788bfcbc-kube-api-access-qh9vw\") on node \"ip-10-0-142-5.ec2.internal\" DevicePath \"\""
Jul 09 04:41:37.444719 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:37.444695    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed25118818b16ec07677f734af92803e66a41a24390df7c39524e2460a91778f"
Jul 09 04:41:37.444719 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:37.444698    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7530-140361441258352"
Jul 09 04:41:37.456145 ip-10-0-142-5 kubenswrapper[2558]: E0709 04:41:37.456109    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7530-140361441258352\" is forbidden: User \"system:node:ip-10-0-142-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-5.ec2.internal' and this object" podUID="c0422ce4-af95-4350-b3d1-bb20788bfcbc" pod="openshift-ingress/test-curl-7530-140361441258352"
Jul 09 04:41:38.110027 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:38.109993    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0422ce4-af95-4350-b3d1-bb20788bfcbc" path="/var/lib/kubelet/pods/c0422ce4-af95-4350-b3d1-bb20788bfcbc/volumes"
Jul 09 04:41:41.177869 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:41.177837    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:41:53.787501 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:53.787420    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9mrbc"]
Jul 09 04:41:55.928365 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:55.928328    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2l5z7_5fd4763c-f863-46ec-a3d9-2b75a89890ff/manager/0.log"
Jul 09 04:41:56.034768 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:56.034739    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df844f965-59c7h_888128d3-c724-47a8-b1fd-a0277e8c028c/copy-manifests/0.log"
Jul 09 04:41:56.053143 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:56.053112    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df844f965-59c7h_888128d3-c724-47a8-b1fd-a0277e8c028c/manager/0.log"
Jul 09 04:41:58.268414 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:58.268387    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-42zx9_3b76c957-042f-4545-b948-23071e7bc1ce/kuadrant-console-plugin/0.log"
Jul 09 04:41:58.606513 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:58.606428    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9mrbc_a8e8a32c-9a5f-4b18-ac4b-e451f4f98399/limitador/0.log"
Jul 09 04:41:59.383192 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:59.383163    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779b7dc9c4-shdwq_09b427f1-f906-43a0-83fe-99fb1646ef7b/kube-auth-proxy/0.log"
Jul 09 04:41:59.857058 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:41:59.857026    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fd7cc5c49-nblmz_f1c2b0f3-4be4-4e34-975f-ced29f667c3d/router/0.log"
Jul 09 04:42:00.538315 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:00.538281    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2qvwx_a3521141-c52e-41c5-a9c6-0645a65cf678/main/0.log"
Jul 09 04:42:00.759552 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:00.759521    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-rt8m6_56b63dba-264b-460c-808f-54af0557d23c/main/0.log"
Jul 09 04:42:07.896368 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:07.896336    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4f6rf_ced5b3ab-d85d-457c-890b-26f1e50b6059/global-pull-secret-syncer/0.log"
Jul 09 04:42:08.074771 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:08.074744    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gjl8b_94d7622f-22d9-4229-b344-78fd728ed2d7/konnectivity-agent/0.log"
Jul 09 04:42:08.187442 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:08.187357    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-5.ec2.internal_cff16d302c0c6fa07ce53eea472441e3/haproxy/0.log"
Jul 09 04:42:13.337120 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:13.337094    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-42zx9_3b76c957-042f-4545-b948-23071e7bc1ce/kuadrant-console-plugin/0.log"
Jul 09 04:42:13.598424 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:13.598343    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9mrbc_a8e8a32c-9a5f-4b18-ac4b-e451f4f98399/limitador/0.log"
Jul 09 04:42:15.792041 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:15.792011    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-66964b95c8-4xbjf_35bd7202-1875-4260-a2e7-00e0f2efc933/metrics-server/0.log"
Jul 09 04:42:16.018612 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.018577    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wksm9_00ee2827-a243-4364-bb8c-81d250338e94/node-exporter/0.log"
Jul 09 04:42:16.037944 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.037915    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wksm9_00ee2827-a243-4364-bb8c-81d250338e94/kube-rbac-proxy/0.log"
Jul 09 04:42:16.060398 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.060329    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wksm9_00ee2827-a243-4364-bb8c-81d250338e94/init-textfile/0.log"
Jul 09 04:42:16.367016 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.366929    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-mh28d_2414171f-da52-4805-b5c0-65e2a3734654/prometheus-operator/0.log"
Jul 09 04:42:16.385402 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.385374    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-mh28d_2414171f-da52-4805-b5c0-65e2a3734654/kube-rbac-proxy/0.log"
Jul 09 04:42:16.410711 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.410680    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-7rh4z_4c7adf19-91ea-4e32-8de8-a9f2ca6fcb7f/prometheus-operator-admission-webhook/0.log"
Jul 09 04:42:16.535651 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.535621    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66985f7865-wbqlg_3719dd95-45b2-4870-be21-4099d0f1b028/thanos-query/0.log"
Jul 09 04:42:16.560162 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.560133    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66985f7865-wbqlg_3719dd95-45b2-4870-be21-4099d0f1b028/kube-rbac-proxy-web/0.log"
Jul 09 04:42:16.585636 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.585610    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66985f7865-wbqlg_3719dd95-45b2-4870-be21-4099d0f1b028/kube-rbac-proxy/0.log"
Jul 09 04:42:16.618453 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.618379    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66985f7865-wbqlg_3719dd95-45b2-4870-be21-4099d0f1b028/prom-label-proxy/0.log"
Jul 09 04:42:16.644009 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.643980    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66985f7865-wbqlg_3719dd95-45b2-4870-be21-4099d0f1b028/kube-rbac-proxy-rules/0.log"
Jul 09 04:42:16.670453 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:16.670427    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66985f7865-wbqlg_3719dd95-45b2-4870-be21-4099d0f1b028/kube-rbac-proxy-metrics/0.log"
Jul 09 04:42:17.963563 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.963534    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"]
Jul 09 04:42:17.963933 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.963872    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0422ce4-af95-4350-b3d1-bb20788bfcbc" containerName="test-curl-7530-140361441258352"
Jul 09 04:42:17.963933 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.963884    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0422ce4-af95-4350-b3d1-bb20788bfcbc" containerName="test-curl-7530-140361441258352"
Jul 09 04:42:17.964011 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.963959    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0422ce4-af95-4350-b3d1-bb20788bfcbc" containerName="test-curl-7530-140361441258352"
Jul 09 04:42:17.969343 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.969320    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:17.971842 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.971810    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v8skv\"/\"default-dockercfg-92sp8\""
Jul 09 04:42:17.972046 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.972024    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v8skv\"/\"openshift-service-ca.crt\""
Jul 09 04:42:17.972919 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.972899    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v8skv\"/\"kube-root-ca.crt\""
Jul 09 04:42:17.974760 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.974738    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"]
Jul 09 04:42:17.996803 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.996772    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-podres\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:17.996976 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.996823    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-sys\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:17.996976 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.996878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-proc\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:17.996976 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.996911    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxsp\" (UniqueName: \"kubernetes.io/projected/6b3fa34a-e25e-4d68-b772-93b496a53587-kube-api-access-lpxsp\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:17.996976 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:17.996928    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-lib-modules\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.098313 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.098278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-proc\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.098487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.098330    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxsp\" (UniqueName: \"kubernetes.io/projected/6b3fa34a-e25e-4d68-b772-93b496a53587-kube-api-access-lpxsp\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.098487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.098358    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-lib-modules\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.098487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.098411    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-proc\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.098487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.098429    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-podres\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.098487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.098486    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-sys\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.098674 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.098539    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-podres\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.098674 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.098567    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-sys\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.098674 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.098584    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b3fa34a-e25e-4d68-b772-93b496a53587-lib-modules\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.107848 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.107818    2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxsp\" (UniqueName: \"kubernetes.io/projected/6b3fa34a-e25e-4d68-b772-93b496a53587-kube-api-access-lpxsp\") pod \"perf-node-gather-daemonset-7g7db\" (UID: \"6b3fa34a-e25e-4d68-b772-93b496a53587\") " pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.280532 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.280439    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.419132 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.419108    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"]
Jul 09 04:42:18.421534 ip-10-0-142-5 kubenswrapper[2558]: W0709 04:42:18.421505    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b3fa34a_e25e_4d68_b772_93b496a53587.slice/crio-a76b84ea69f795994d517ca6f699bf3bb01bc1e8018b9da0053cea649c13cca7 WatchSource:0}: Error finding container a76b84ea69f795994d517ca6f699bf3bb01bc1e8018b9da0053cea649c13cca7: Status 404 returned error can't find the container with id a76b84ea69f795994d517ca6f699bf3bb01bc1e8018b9da0053cea649c13cca7
Jul 09 04:42:18.637282 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.637238    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db" event={"ID":"6b3fa34a-e25e-4d68-b772-93b496a53587","Type":"ContainerStarted","Data":"079e6d9cba0746cb23a8835d50181e7455bd24daaa17d49fbfe24f28ca65f672"}
Jul 09 04:42:18.637471 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.637304    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db" event={"ID":"6b3fa34a-e25e-4d68-b772-93b496a53587","Type":"ContainerStarted","Data":"a76b84ea69f795994d517ca6f699bf3bb01bc1e8018b9da0053cea649c13cca7"}
Jul 09 04:42:18.637471 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.637357    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:18.654368 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.654316    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db" podStartSLOduration=1.654301217 podStartE2EDuration="1.654301217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 04:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 04:42:18.652744417 +0000 UTC m=+2545.027748236" watchObservedRunningTime="2026-07-09 04:42:18.654301217 +0000 UTC m=+2545.029305073"
Jul 09 04:42:18.985062 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:18.985035    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6887f95f6-298l5_f52501a6-04a1-4a4f-af27-b87d91f49946/console/0.log"
Jul 09 04:42:20.602081 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:20.602053    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ndcnh_6da771dc-17ac-4f66-9276-1b2fda78a9dd/dns/0.log"
Jul 09 04:42:20.623181 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:20.623154    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ndcnh_6da771dc-17ac-4f66-9276-1b2fda78a9dd/kube-rbac-proxy/0.log"
Jul 09 04:42:20.735267 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:20.735241    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6v76_72e5e84b-cb3b-4a19-a75c-a1150fd26a50/dns-node-resolver/0.log"
Jul 09 04:42:21.323838 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:21.323807    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vvqj2_d5d7965a-ab05-4758-997b-883c2e788152/node-ca/0.log"
Jul 09 04:42:22.536090 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:22.536058    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779b7dc9c4-shdwq_09b427f1-f906-43a0-83fe-99fb1646ef7b/kube-auth-proxy/0.log"
Jul 09 04:42:22.785487 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:22.785461    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fd7cc5c49-nblmz_f1c2b0f3-4be4-4e34-975f-ced29f667c3d/router/0.log"
Jul 09 04:42:23.560240 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:23.560163    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xqjt6_1cc1c1e6-f43e-48f9-8488-9ab9c9da5093/serve-healthcheck-canary/0.log"
Jul 09 04:42:24.071844 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:24.071817    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6z7dk_17a511ac-2f5c-4fc2-904c-ffeb76048e4c/kube-rbac-proxy/0.log"
Jul 09 04:42:24.102484 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:24.102456    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6z7dk_17a511ac-2f5c-4fc2-904c-ffeb76048e4c/exporter/0.log"
Jul 09 04:42:24.133177 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:24.133135    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6z7dk_17a511ac-2f5c-4fc2-904c-ffeb76048e4c/extractor/0.log"
Jul 09 04:42:24.652641 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:24.652615    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v8skv/perf-node-gather-daemonset-7g7db"
Jul 09 04:42:26.873968 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:26.873929    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2l5z7_5fd4763c-f863-46ec-a3d9-2b75a89890ff/manager/0.log"
Jul 09 04:42:27.003018 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:27.002985    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df844f965-59c7h_888128d3-c724-47a8-b1fd-a0277e8c028c/manager/0.log"
Jul 09 04:42:27.023914 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:27.023888    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df844f965-59c7h_888128d3-c724-47a8-b1fd-a0277e8c028c/copy-manifests/0.log"
Jul 09 04:42:28.111817 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:28.111786    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779b7dc9c4-shdwq_09b427f1-f906-43a0-83fe-99fb1646ef7b/kube-auth-proxy/0.log"
Jul 09 04:42:28.303879 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:28.303797    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fd7cc5c49-nblmz_f1c2b0f3-4be4-4e34-975f-ced29f667c3d/router/0.log"
Jul 09 04:42:28.481604 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:28.481573    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-765cd6b57c-lcvtc_5b158fc3-0386-47b2-8159-f583fbd7a1b9/manager/0.log"
Jul 09 04:42:28.531710 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:28.531673    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-p79j8_9c3fbdb0-639e-44a5-88e3-043bf26e3471/openshift-lws-operator/0.log"
Jul 09 04:42:29.905810 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:29.905781    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779b7dc9c4-shdwq_09b427f1-f906-43a0-83fe-99fb1646ef7b/kube-auth-proxy/0.log"
Jul 09 04:42:30.108098 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:30.108006    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fd7cc5c49-nblmz_f1c2b0f3-4be4-4e34-975f-ced29f667c3d/router/0.log"
Jul 09 04:42:33.299909 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:33.299880    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-g9qrb_c518becb-972d-4285-957d-09084364aee5/migrator/0.log"
Jul 09 04:42:33.319604 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:33.319577    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-g9qrb_c518becb-972d-4285-957d-09084364aee5/graceful-termination/0.log"
Jul 09 04:42:33.662987 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:33.662955    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-cfhxq_66835b46-56de-45ff-9bf2-997eb83836f5/kube-storage-version-migrator-operator/1.log"
Jul 09 04:42:33.664715 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:33.664687    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-cfhxq_66835b46-56de-45ff-9bf2-997eb83836f5/kube-storage-version-migrator-operator/0.log"
Jul 09 04:42:34.730872 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:34.730843    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6rnsk_5645ae66-7358-4067-b40c-3c0d5322f0f0/kube-multus/0.log"
Jul 09 04:42:35.065957 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:35.065927    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xbcdx_2bc32fee-2e66-4598-866b-2a22de571626/kube-multus-additional-cni-plugins/0.log"
Jul 09 04:42:35.086677 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:35.086652    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xbcdx_2bc32fee-2e66-4598-866b-2a22de571626/egress-router-binary-copy/0.log"
Jul 09 04:42:35.106267 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:35.106240    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xbcdx_2bc32fee-2e66-4598-866b-2a22de571626/cni-plugins/0.log"
Jul 09 04:42:35.126995 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:35.126973    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xbcdx_2bc32fee-2e66-4598-866b-2a22de571626/bond-cni-plugin/0.log"
Jul 09 04:42:35.146858 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:35.146835    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xbcdx_2bc32fee-2e66-4598-866b-2a22de571626/routeoverride-cni/0.log"
Jul 09 04:42:35.167915 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:35.167888    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xbcdx_2bc32fee-2e66-4598-866b-2a22de571626/whereabouts-cni-bincopy/0.log"
Jul 09 04:42:35.188442 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:35.188419    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xbcdx_2bc32fee-2e66-4598-866b-2a22de571626/whereabouts-cni/0.log"
Jul 09 04:42:35.326059 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:35.325964    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h7zph_29ca9faa-1fb7-4223-a246-f01696f726e4/network-metrics-daemon/0.log"
Jul 09 04:42:35.344173 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:35.344145    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h7zph_29ca9faa-1fb7-4223-a246-f01696f726e4/kube-rbac-proxy/0.log"
Jul 09 04:42:36.413238 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:36.413190    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg587_c0578c1a-c573-4cbd-9d2f-9f30f0954753/ovn-controller/0.log"
Jul 09 04:42:36.437542 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:36.437503    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg587_c0578c1a-c573-4cbd-9d2f-9f30f0954753/ovn-acl-logging/0.log"
Jul 09 04:42:36.464197 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:36.464166    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg587_c0578c1a-c573-4cbd-9d2f-9f30f0954753/kube-rbac-proxy-node/0.log"
Jul 09 04:42:36.486998 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:36.486972    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg587_c0578c1a-c573-4cbd-9d2f-9f30f0954753/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 04:42:36.504436 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:36.504409    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg587_c0578c1a-c573-4cbd-9d2f-9f30f0954753/northd/0.log"
Jul 09 04:42:36.524795 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:36.524764    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg587_c0578c1a-c573-4cbd-9d2f-9f30f0954753/nbdb/0.log"
Jul 09 04:42:36.545539 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:36.545507    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg587_c0578c1a-c573-4cbd-9d2f-9f30f0954753/sbdb/0.log"
Jul 09 04:42:36.626087 ip-10-0-142-5 kubenswrapper[2558]: I0709 04:42:36.626044    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg587_c0578c1a-c573-4cbd-9d2f-9f30f0954753/ovnkube-controller/0.log"