Jul 24 15:40:38.915810 ip-10-0-131-210 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 15:40:39.361905 ip-10-0-131-210 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 15:40:39.361905 ip-10-0-131-210 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 15:40:39.361905 ip-10-0-131-210 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 15:40:39.361905 ip-10-0-131-210 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 15:40:39.361905 ip-10-0-131-210 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 15:40:39.364969 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.364832    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 15:40:39.369960 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369945    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:40:39.369960 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369959    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369963    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369968    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369971    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369975    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369977    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369980    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369982    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369985    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369987    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369996    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.369999    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370002    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370004    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370007    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370011    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370015    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370018    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370021    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:40:39.370027 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370023    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370026    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370029    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370032    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370036    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370039    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370042    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370044    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370047    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370049    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370052    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370054    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370057    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370059    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370061    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370064    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370066    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370068    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370071    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:40:39.370481 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370073    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370076    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370079    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370081    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370085    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370087    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370090    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370092    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370094    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370096    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370099    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370102    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370104    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370106    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370110    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370116    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370119    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370121    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370124    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:40:39.371087 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370126    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370129    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370131    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370134    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370136    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370139    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370141    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370146    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370149    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370152    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370154    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370157    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370159    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370162    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370164    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370166    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370169    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370171    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370174    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:40:39.371770 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370177    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370179    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370181    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370184    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370187    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370189    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370191    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370193    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370196    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370199    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370202    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370204    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370206    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370209    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370211    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370213    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370216    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370218    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370221    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370223    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:40:39.372261 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370226    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.370228    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372431    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372457    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372461    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372464    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372467    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372469    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372472    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372474    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372477    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372479    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372481    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372484    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372487    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372489    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372497    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372500    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372503    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372506    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:40:39.372748 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372508    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372511    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372513    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372516    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372518    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372520    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372523    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372526    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372531    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372534    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372536    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372539    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372541    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372543    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372546    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372549    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372552    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372554    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372557    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372559    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:40:39.373258 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372562    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372564    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372566    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372569    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372571    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372573    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372576    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372579    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372581    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372584    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372586    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372588    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372591    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372593    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372596    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372598    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372600    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372603    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372606    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372608    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:40:39.373798 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372611    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372613    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372615    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372620    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372625    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372628    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372630    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372634    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372637    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372641    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372644    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372647    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372650    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372653    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372655    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372658    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372661    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372663    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372665    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:40:39.374278 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372668    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372670    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372674    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372676    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372678    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372681    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372684    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372686    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372689    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372691    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372693    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372696    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372699    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372702    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372706    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372709    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372712    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372714    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372717    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:40:39.374745 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372719    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372723    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.372731    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373149    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373155    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373157    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373160    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373163    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373166    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373168    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373171    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373174    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373176    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373178    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373181    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373184    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373186    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373189    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373193    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373197    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:40:39.375210 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373200    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373202    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373205    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373208    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373211    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373213    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373216    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373219    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373221    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373224    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373226    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373228    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373231    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373234    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373236    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373239    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373241    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373244    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373246    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:40:39.375704 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373249    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373251    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373253    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373256    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373258    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373261    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373263    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373266    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373268    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373272    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373275    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373277    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373280    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373283    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373285    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373287    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373290    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373293    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373295    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373298    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:40:39.376166 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373300    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373303    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373305    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373308    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373310    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373313    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373315    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373318    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373320    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373322    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373325    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373327    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373330    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373332    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373335    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373337    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373340    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373342    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373345    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373347    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:40:39.376681 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373356    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373360    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373363    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373366    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373368    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373371    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373374    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373377    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373380    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373382    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373385    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373387    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373391    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373394    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373397    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373400    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373402    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373405    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373408    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373411    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:40:39.377174 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373414    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373416    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.373419    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373502    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373509    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373515    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373519    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373523    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373526    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373531    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373535    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373538    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373541    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373544    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373547    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373551    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373553    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373556    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373559    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373562    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373566    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373573    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373576    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 15:40:39.377662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373579    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373582    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373585    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373588    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373591    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373594    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373597    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373600    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373603    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373605    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373608    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373611    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373615    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373618    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373621    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373623    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373626    2576 flags.go:64] FLAG: --enable-server="true"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373629    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373633    2576 flags.go:64] FLAG: --event-burst="100"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373636    2576 flags.go:64] FLAG: --event-qps="50"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373638    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373642    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373644    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373648    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 15:40:39.378245 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373651    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373654    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373657    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373660    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373663    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373667    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373672    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373674    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373677    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373680    2576 flags.go:64] FLAG: --feature-gates=""
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373683    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373686    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373689    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373692    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373695    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373698    2576 flags.go:64] FLAG: --help="false"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373700    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373703    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373706    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373709    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373712    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373715    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373718    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 15:40:39.378815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373721    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373723    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373726    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373728    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373731    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373734    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373737    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373739    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373742    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373744    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373747    2576 flags.go:64] FLAG: --lock-file=""
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373750    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373753    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373755    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373762    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373766    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373769    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373772    2576 flags.go:64] FLAG: --logging-format="text"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373774    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373778    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373781    2576 flags.go:64] FLAG: --manifest-url=""
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373783    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373787    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373790    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373794    2576 flags.go:64] FLAG: --max-pods="110"
Jul 24 15:40:39.379383 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373797    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373800    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373803    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373806    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373809    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373812    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373814    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373821    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373824    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373827    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373830    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373833    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373838    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373841    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373844    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373846    2576 flags.go:64] FLAG: --port="10250"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373849    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373852    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b3affcb2569997b7"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373855    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373869    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373873    2576 flags.go:64] FLAG: --register-node="true"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373876    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373880    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373886    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 24 15:40:39.380022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373889    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373891    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373895    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373898    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373901    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373903    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373906    2576 flags.go:64] FLAG: --runonce="false"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373909    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373911    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373914    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373917    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373919    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373922    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373925    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373928    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373931    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373933    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373936    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373939    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373941    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373944    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373947    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373952    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373955    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373958    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 15:40:39.380593 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373962    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373964    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373967    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373970    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373973    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373976    2576 flags.go:64] FLAG: --v="2"
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373981    2576 flags.go:64] FLAG: --version="false"
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373990    2576 flags.go:64] FLAG: --vmodule=""
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373993    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.373996    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374142    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374147    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374150    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374153    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374156    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374158    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374161    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374164    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374166    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374169    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374171    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374174    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:40:39.381249 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374177    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374179    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374181    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374184    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374186    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374189    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374192    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374196    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374198    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374201    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374204    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374206    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374209    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374211    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374215    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374217    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374221    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374223    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374226    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:40:39.381838 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374229    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374232    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374235    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374237    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374239    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374242    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374244    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374246    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374249    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374251    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374254    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374256    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374258    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374261    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374263    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374265    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374268    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374271    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374273    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374275    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:40:39.382308 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374278    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374280    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374283    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374285    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374288    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374290    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374292    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374297    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374299    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374303    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374305    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374308    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374310    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374313    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374315    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374317    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374320    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374324    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374328    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374331    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:40:39.382802 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374333    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374335    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374339    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374342    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374345    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374347    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374350    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374352    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374355    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374358    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374360    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374362    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374365    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374372    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374374    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374376    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374379    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374381    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374383    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:40:39.383263 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374387    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374389    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374393    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374395    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374398    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374400    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374403    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374406    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.374408    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.375123    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.382430    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.382465    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.383385    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.383393    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383468    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:40:39.383776 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383474    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383478    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383481    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383484    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383486    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383489    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383492    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383494    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383497    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383500    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383502    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383505    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383508    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383510    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383513    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383515    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383518    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383520    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383522    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383525    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:40:39.384142 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383527    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383529    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383532    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383534    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383537    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383539    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383541    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383544    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383548    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383551    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383553    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383557    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383561    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383564    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383566    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383569    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383571    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383573    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383576    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:40:39.384620 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383578    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383581    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383583    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383585    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383588    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383590    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383593    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383595    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383597    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383600    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383602    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383605    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383607    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383609    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383612    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383614    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383617    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383619    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383622    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383624    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:40:39.385078 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383627    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383630    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383634    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383636    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383639    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383641    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383643    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383646    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383648    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383651    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383653    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383656    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383658    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383660    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383663    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383665    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383667    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383671    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383675    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:40:39.385587 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383678    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383680    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383683    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383685    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383689    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383691    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383694    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383696    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383698    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383701    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383703    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383706    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383709    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383711    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383714    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383716    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383719    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383721    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383724    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:40:39.386049 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383727    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.383732    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383848    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383852    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383855    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383858    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383861    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383864    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383867    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383870    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383873    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383875    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383878    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383880    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383883    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 15:40:39.386525 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383885    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383888    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383890    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383893    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383895    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383898    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383900    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383903    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383905    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383907    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383910    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383914    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383917    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383920    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383923    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383926    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383929    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383931    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383934    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 15:40:39.386888 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383936    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383939    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383941    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383943    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383945    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383948    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383951    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383955    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383957    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383959    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383962    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383964    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383966    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383969    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383972    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383975    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383977    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383979    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383982    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383984    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 15:40:39.387322 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383986    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383988    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383991    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383994    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383996    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.383998    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384001    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384003    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384006    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384008    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384011    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384013    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384015    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384018    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384020    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384022    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384025    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384027    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384029    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384032    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 15:40:39.387850 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384034    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384036    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384039    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384042    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384044    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384046    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384049    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384051    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384054    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384056    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384058    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384061    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384063    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384065    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384068    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384070    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384073    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384075    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384077    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384080    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 15:40:39.388313 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384082    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 15:40:39.388831 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384085    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 15:40:39.388831 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384088    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 15:40:39.388831 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384091    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 15:40:39.388831 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384094    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 15:40:39.388831 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384097    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 15:40:39.388831 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:39.384100    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 15:40:39.388831 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.384104    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 15:40:39.388831 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.384228    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 15:40:39.388831 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.388598    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 15:40:39.390282 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.390267    2576 server.go:1019] "Starting client certificate rotation"
Jul 24 15:40:39.390397 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.390378    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 15:40:39.390470 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.390425    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 15:40:39.393105 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.393089    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 15:40:39.394287 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.394267    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 15:40:39.405885 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.405852    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 24 15:40:39.408494 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.408477    2576 log.go:25] "Validated CRI v1 image API"
Jul 24 15:40:39.410140 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.410125    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 15:40:39.412923 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.412894    2576 fs.go:135] Filesystem UUIDs: map[3947add7-38c6-4f8b-a802-39d6f7dfff63:/dev/nvme0n1p4 3e2fc0e8-0bc7-47ef-86f8-fff06ead4b12:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 24 15:40:39.412998 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.412921    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 15:40:39.419196 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.419078    2576 manager.go:216] Machine: {Timestamp:2026-07-24 15:40:39.417607027 +0000 UTC m=+0.371966446 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3249340 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2422f8d09dcbbe31a88641136f9f75 SystemUUID:ec2422f8-d09d-cbbe-31a8-8641136f9f75 BootID:e247213e-dfbc-46ff-a3cc-86d35fa36ac5 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5b:31:90:f0:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5b:31:90:f0:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:f4:f8:49:fe:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 15:40:39.419196 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.419194    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 15:40:39.419305 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.419278    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 15:40:39.422656 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.422623    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 15:40:39.422806 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.422659    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 15:40:39.423387 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.423372    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 15:40:39.423638 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.423626    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 15:40:39.423719 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.423640    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 15:40:39.423719 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.423653    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 15:40:39.423719 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.423669    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 15:40:39.424929 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.424917    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 15:40:39.426807 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.426795    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 15:40:39.426924 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.426915    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 15:40:39.426959 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.426946    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 15:40:39.426989 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.426958    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 15:40:39.426989 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.426973    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 15:40:39.426989 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.426983    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 24 15:40:39.427062 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.426991    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 15:40:39.427362 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.427351    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 15:40:39.427404 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.427370    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 15:40:39.429856 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.429836    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 15:40:39.432308 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.432293    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 15:40:39.432388 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.432330    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 15:40:39.434539 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434525    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434544    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434553    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434560    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434572    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434580    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434591    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434599    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434609    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434617    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 15:40:39.434626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434629    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 15:40:39.434941 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.434638    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 15:40:39.435510 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.435498    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 15:40:39.435556 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.435513    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 15:40:39.438928 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.438907    2576 server.go:1268] "Started kubelet"
Jul 24 15:40:39.439113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.439009    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 15:40:39.439113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.439023    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 15:40:39.439113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.439057    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 15:40:39.439280 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.439118    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 15:40:39.439327 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.439274    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 15:40:39.439375 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.439037    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 15:40:39.439817 ip-10-0-131-210 systemd[1]: Started Kubernetes Kubelet.
Jul 24 15:40:39.445743 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.445714    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 15:40:39.446497 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.446474    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 15:40:39.446811 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.446789    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 15:40:39.452000 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.450889    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-210.ec2.internal.18c5440a8da2b5ff  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-210.ec2.internal,UID:ip-10-0-131-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-210.ec2.internal,},FirstTimestamp:2026-07-24 15:40:39.438882303 +0000 UTC m=+0.393241722,LastTimestamp:2026-07-24 15:40:39.438882303 +0000 UTC m=+0.393241722,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-210.ec2.internal,}"
Jul 24 15:40:39.455253 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.455233    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 15:40:39.455341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.455319    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 15:40:39.455462 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.455416    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:39.456178 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456159    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 15:40:39.456178 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456169    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 15:40:39.456300 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456184    2576 factory.go:55] Registering systemd factory
Jul 24 15:40:39.456300 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456192    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 24 15:40:39.456300 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456160    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 15:40:39.456300 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456244    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 15:40:39.456477 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456328    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 15:40:39.456477 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456339    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 15:40:39.456477 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456431    2576 factory.go:153] Registering CRI-O factory
Jul 24 15:40:39.456477 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456460    2576 factory.go:223] Registration of the crio container factory successfully
Jul 24 15:40:39.456625 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456491    2576 factory.go:103] Registering Raw factory
Jul 24 15:40:39.456625 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.456510    2576 manager.go:1196] Started watching for new ooms in manager
Jul 24 15:40:39.457251 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.457117    2576 manager.go:349] Starting recovery of all containers
Jul 24 15:40:39.458752 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.458713    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 15:40:39.458842 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.458804    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 15:40:39.466193 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.466177    2576 manager.go:354] Recovery completed
Jul 24 15:40:39.469072 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.469048    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j2wbr"
Jul 24 15:40:39.470289 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.470273    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:40:39.472234 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.472219    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:40:39.472304 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.472252    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:40:39.472304 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.472264    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:40:39.472859 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.472839    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 15:40:39.472859 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.472853    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:40:39.472998 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.472857    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 15:40:39.472998 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.472938    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 15:40:39.474009 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.473994    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:40:39.474080 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.474021    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:40:39.474080 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.474033    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:40:39.474912 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.474898    2576 policy_none.go:49] "None policy: Start"
Jul 24 15:40:39.474963 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.474915    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 15:40:39.474963 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.474926    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 15:40:39.475937 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.475926    2576 policy_none.go:47] "Start"
Jul 24 15:40:39.476529 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.476516    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j2wbr"
Jul 24 15:40:39.483425 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.483240    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-210.ec2.internal.18c5440a8f9fa569  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-210.ec2.internal,UID:ip-10-0-131-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-210.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-210.ec2.internal,},FirstTimestamp:2026-07-24 15:40:39.472235881 +0000 UTC m=+0.426595298,LastTimestamp:2026-07-24 15:40:39.472235881 +0000 UTC m=+0.426595298,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-210.ec2.internal,}"
Jul 24 15:40:39.510132 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.510111    2576 manager.go:337] "Starting Device Plugin manager"
Jul 24 15:40:39.510259 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.510149    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 15:40:39.510259 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.510161    2576 server.go:87] "Starting device plugin registration server"
Jul 24 15:40:39.510491 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.510479    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 15:40:39.510559 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.510495    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 15:40:39.510629 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.510613    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 15:40:39.510732 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.510702    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 15:40:39.510732 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.510716    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 15:40:39.511243 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.511220    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 15:40:39.511332 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.511279    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:39.527548 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.527514    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 15:40:39.532410 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.528978    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 15:40:39.532410 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.529017    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 15:40:39.532410 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.529037    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 15:40:39.532410 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.529072    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 15:40:39.532410 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.531074    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:40:39.611634 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.611597    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:40:39.612603 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.612553    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:40:39.612603 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.612590    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:40:39.612706 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.612606    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:40:39.612706 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.612631    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.622155 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.622136    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.622226 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.622163    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-210.ec2.internal\": node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:39.629697 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.629679    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"]
Jul 24 15:40:39.629787 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.629757    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:40:39.630620 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.630603    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:40:39.630691 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.630631    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:40:39.630691 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.630641    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:40:39.632871 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.632857    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:40:39.632996 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.632981    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.633036 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.633012    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:40:39.633550 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.633534    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:40:39.633611 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.633568    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:40:39.633611 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.633579    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:40:39.633681 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.633541    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:40:39.633681 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.633632    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:40:39.633681 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.633645    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:40:39.635899 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.635883    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.635967 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.635915    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:40:39.636643 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.636630    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:40:39.636704 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.636657    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:40:39.636704 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.636669    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:40:39.648076 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.648052    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:39.650036 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.650022    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-210.ec2.internal\" not found" node="ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.653547 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.653533    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-210.ec2.internal\" not found" node="ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.657713 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.657693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43b6e5fc9d1a935acaab964e139534e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"43b6e5fc9d1a935acaab964e139534e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.657809 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.657719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6924941272464cf155cde721c0cf91a1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-210.ec2.internal\" (UID: \"6924941272464cf155cde721c0cf91a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.657809 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.657737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43b6e5fc9d1a935acaab964e139534e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"43b6e5fc9d1a935acaab964e139534e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.748600 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.748566    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:39.757970 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.757941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43b6e5fc9d1a935acaab964e139534e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"43b6e5fc9d1a935acaab964e139534e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.758039 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.757975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43b6e5fc9d1a935acaab964e139534e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"43b6e5fc9d1a935acaab964e139534e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.758039 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.757991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6924941272464cf155cde721c0cf91a1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-210.ec2.internal\" (UID: \"6924941272464cf155cde721c0cf91a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.758099 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.758039    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43b6e5fc9d1a935acaab964e139534e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"43b6e5fc9d1a935acaab964e139534e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.758099 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.758050    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43b6e5fc9d1a935acaab964e139534e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"43b6e5fc9d1a935acaab964e139534e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.758099 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.758081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6924941272464cf155cde721c0cf91a1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-210.ec2.internal\" (UID: \"6924941272464cf155cde721c0cf91a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.849170 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.849127    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:39.949895 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:39.949806    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:39.953986 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.953968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:39.957796 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:39.957780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:40.050891 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:40.050849    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:40.151427 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:40.151389    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:40.252036 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:40.251954    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:40.352577 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:40.352537    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:40.389112 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.389080    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 15:40:40.389609 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.389285    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 15:40:40.453500 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:40.453462    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:40.481945 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.481910    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 15:35:40 +0000 UTC" deadline="2028-04-12 06:04:52.90848644 +0000 UTC"
Jul 24 15:40:40.481945 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.481946    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15062h24m12.426545538s"
Jul 24 15:40:40.554430 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:40.554363    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:40.603325 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:40.603285    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6924941272464cf155cde721c0cf91a1.slice/crio-f8247c54ebd3ac6dfb628921237f91d865cfa71825b14af248376dc2e5a2a7e7 WatchSource:0}: Error finding container f8247c54ebd3ac6dfb628921237f91d865cfa71825b14af248376dc2e5a2a7e7: Status 404 returned error can't find the container with id f8247c54ebd3ac6dfb628921237f91d865cfa71825b14af248376dc2e5a2a7e7
Jul 24 15:40:40.603651 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:40.603630    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b6e5fc9d1a935acaab964e139534e4.slice/crio-03a0e89055216420a32056e3d0b24aacc0aabe4d293a358d7c227be8f3b90140 WatchSource:0}: Error finding container 03a0e89055216420a32056e3d0b24aacc0aabe4d293a358d7c227be8f3b90140: Status 404 returned error can't find the container with id 03a0e89055216420a32056e3d0b24aacc0aabe4d293a358d7c227be8f3b90140
Jul 24 15:40:40.608912 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.608898    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:40:40.613568 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.613549    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 15:40:40.614647 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.614635    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 15:40:40.614713 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.614659    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 15:40:40.614713 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.614668    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 24 15:40:40.652580 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.652549    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:40:40.654672 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:40.654656    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:40.755332 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:40.755302    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:40.855795 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:40.855724    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 24 15:40:40.945826 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.945798    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:40:40.946423 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.946409    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:40:40.956418 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.956402    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:40.970117 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.970099    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 15:40:40.971661 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:40.971649    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 24 15:40:42.449659 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.449560    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 15:40:42.897256 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.897213    2576 apiserver.go:52] "Watching apiserver"
Jul 24 15:40:42.905748 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.905717    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 15:40:42.906173 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.906152    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-wk55j","openshift-multus/network-metrics-daemon-2wl2t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2","openshift-cluster-node-tuning-operator/tuned-5hqlm","openshift-multus/multus-tdm2k","openshift-network-diagnostics/network-check-target-kr79g","openshift-network-operator/iptables-alerter-q8dh5","openshift-ovn-kubernetes/ovnkube-node-mf4lp","kube-system/global-pull-secret-syncer-9k7ms","kube-system/konnectivity-agent-rfc4r","kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal","openshift-dns/node-resolver-xgs2m","openshift-image-registry/node-ca-vf9gg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"]
Jul 24 15:40:42.909308 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.909117    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:42.911734 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.911349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:42.911734 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.911658    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:40:42.912675 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.912464    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hj2m4\""
Jul 24 15:40:42.912675 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.912492    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 15:40:42.912675 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.912610    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 15:40:42.914262 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.914229    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-576wp\""
Jul 24 15:40:42.914436 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.914421    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 15:40:42.914510 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.914447    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 15:40:42.914728 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.914709    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 15:40:42.914969 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.914951    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:42.917098 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.917079    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 15:40:42.917269 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.917220    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 15:40:42.917269 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.917232    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-47phx\""
Jul 24 15:40:42.917495 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.917476    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 15:40:42.917733 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.917720    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.919819 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.919798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.920393 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.920377    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7mk27\""
Jul 24 15:40:42.920527 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.920513    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:40:42.920596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.920570    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 15:40:42.922056 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.922032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:42.922157 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:42.922132    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:40:42.922442 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.922426    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 15:40:42.922637 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.922619    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 15:40:42.922952 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.922937    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 15:40:42.923038 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.922979    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 15:40:42.923119 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.923107    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xxnrf\""
Jul 24 15:40:42.925019 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.925003    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:42.925111 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:42.925074    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:40:42.926381 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.926365    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 15:40:42.927683 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.927653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:42.929793 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.929752    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:42.929894 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:42.929813    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:40:42.930080 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.930054    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-knbwm\""
Jul 24 15:40:42.930334 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.930316    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 15:40:42.930625 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.930592    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 15:40:42.932200 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.932180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:40:42.934893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.934877    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 15:40:42.935014 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.934995    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 15:40:42.935090 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.934935    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-59nmh\""
Jul 24 15:40:42.936055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.936023    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:42.938173 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-sysctl-conf\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.938306 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-registration-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:42.938306 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-hostroot\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.938306 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5k6\" (UniqueName: \"kubernetes.io/projected/de658cfe-d206-4123-9309-e2fe7309d4e5-kube-api-access-kt5k6\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.938306 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-cnibin\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:42.938306 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:42.938306 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-lib-modules\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a324c808-39af-4084-b96b-628e7899bbb2-etc-tuned\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938335    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a324c808-39af-4084-b96b-628e7899bbb2-tmp\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-conf-dir\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-systemd\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466xx\" (UniqueName: \"kubernetes.io/projected/0ad7bbb9-a8d6-40d9-a962-3ce602311838-kube-api-access-466xx\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-run-netns\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-var-lib-kubelet\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938538    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-daemon-config\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ad7bbb9-a8d6-40d9-a962-3ce602311838-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:42.938657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzj8z\" (UniqueName: \"kubernetes.io/projected/f14cf9e9-32cf-4ce9-9761-16bed40d8a5e-kube-api-access-wzj8z\") pod \"node-ca-vf9gg\" (UID: \"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e\") " pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-run\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-var-lib-cni-bin\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-sys\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-os-release\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-run-k8s-cni-cncf-io\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938787    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-run-multus-certs\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7kw8\" (UniqueName: \"kubernetes.io/projected/825c4047-d1f9-4272-a188-f4b57843b902-kube-api-access-j7kw8\") pod \"iptables-alerter-q8dh5\" (UID: \"825c4047-d1f9-4272-a188-f4b57843b902\") " pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f14cf9e9-32cf-4ce9-9761-16bed40d8a5e-serviceca\") pod \"node-ca-vf9gg\" (UID: \"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e\") " pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-device-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/33d7b0af-08cf-4fa6-9f39-e461402d71f1-dbus\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.938998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/825c4047-d1f9-4272-a188-f4b57843b902-iptables-alerter-script\") pod \"iptables-alerter-q8dh5\" (UID: \"825c4047-d1f9-4272-a188-f4b57843b902\") " pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6m96\" (UniqueName: \"kubernetes.io/projected/a324c808-39af-4084-b96b-628e7899bbb2-kube-api-access-q6m96\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-os-release\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:42.939257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f14cf9e9-32cf-4ce9-9761-16bed40d8a5e-host\") pod \"node-ca-vf9gg\" (UID: \"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e\") " pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqrpd\" (UniqueName: \"kubernetes.io/projected/4c568e24-a209-468a-b73f-7d8a0666559f-kube-api-access-cqrpd\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-modprobe-d\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-var-lib-kubelet\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ad7bbb9-a8d6-40d9-a962-3ce602311838-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/825c4047-d1f9-4272-a188-f4b57843b902-host-slash\") pod \"iptables-alerter-q8dh5\" (UID: \"825c4047-d1f9-4272-a188-f4b57843b902\") " pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-host\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-system-cni-dir\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-etc-kubernetes\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ad7bbb9-a8d6-40d9-a962-3ce602311838-cni-binary-copy\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939346    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9qw9q\""
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939357    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-system-cni-dir\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-sys-fs\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-kubernetes\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-cni-dir\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de658cfe-d206-4123-9309-e2fe7309d4e5-cni-binary-copy\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-socket-dir-parent\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.939916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-socket-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:42.940661 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:42.940661 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/33d7b0af-08cf-4fa6-9f39-e461402d71f1-kubelet-config\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:42.940661 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-sysconfig\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.940661 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-sysctl-d\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:42.940661 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-cnibin\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.940661 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-var-lib-cni-multus\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:42.940661 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.939873    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 15:40:42.941181 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.941156    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 15:40:42.941181 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.941174    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 15:40:42.941359 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.941219    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 15:40:42.941359 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.941170    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 15:40:42.941508 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.941493    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 15:40:42.943627 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.942396    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:42.944686 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.944668    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qw2l\""
Jul 24 15:40:42.945124 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.944901    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 15:40:42.945124 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.944906    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 15:40:42.983735 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.983682    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p8cf9"
Jul 24 15:40:42.989071 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:42.989052    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p8cf9"
Jul 24 15:40:43.004771 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.004710    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal" event={"ID":"43b6e5fc9d1a935acaab964e139534e4","Type":"ContainerStarted","Data":"03a0e89055216420a32056e3d0b24aacc0aabe4d293a358d7c227be8f3b90140"}
Jul 24 15:40:43.006691 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.006663    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal" event={"ID":"6924941272464cf155cde721c0cf91a1","Type":"ContainerStarted","Data":"f8247c54ebd3ac6dfb628921237f91d865cfa71825b14af248376dc2e5a2a7e7"}
Jul 24 15:40:43.027033 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.027007    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 15:40:43.040760 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.040723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-systemd-units\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.040760 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.040764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-log-socket\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.041025 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.040790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-cni-bin\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.041025 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.040832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:43.041025 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.040929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-run-netns\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.041190 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a167781-23cd-4fec-880a-76ccee6874c5-ovnkube-script-lib\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.041190 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-systemd\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.041190 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466xx\" (UniqueName: \"kubernetes.io/projected/0ad7bbb9-a8d6-40d9-a962-3ce602311838-kube-api-access-466xx\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.041190 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-run-netns\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.041190 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-var-lib-kubelet\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.041190 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-daemon-config\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-systemd\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041209    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-run-netns\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041268    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-var-lib-kubelet\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ad7bbb9-a8d6-40d9-a962-3ce602311838-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-etc-openvswitch\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a167781-23cd-4fec-880a-76ccee6874c5-ovnkube-config\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3a39934-0ab7-466e-a958-7ffebde7bb5a-konnectivity-ca\") pod \"konnectivity-agent-rfc4r\" (UID: \"d3a39934-0ab7-466e-a958-7ffebde7bb5a\") " pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzj8z\" (UniqueName: \"kubernetes.io/projected/f14cf9e9-32cf-4ce9-9761-16bed40d8a5e-kube-api-access-wzj8z\") pod \"node-ca-vf9gg\" (UID: \"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e\") " pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-run\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-var-lib-cni-bin\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-var-lib-cni-bin\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.041473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a167781-23cd-4fec-880a-76ccee6874c5-ovn-node-metrics-cert\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642babd4-24ba-47d0-899b-e8345129d52a-tmp-dir\") pod \"node-resolver-xgs2m\" (UID: \"642babd4-24ba-47d0-899b-e8345129d52a\") " pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-sys\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-os-release\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-run\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-run-k8s-cni-cncf-io\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041577    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-run-k8s-cni-cncf-io\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-run-multus-certs\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-sys\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-os-release\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-run-ovn\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041654    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-run-multus-certs\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041690    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7kw8\" (UniqueName: \"kubernetes.io/projected/825c4047-d1f9-4272-a188-f4b57843b902-kube-api-access-j7kw8\") pod \"iptables-alerter-q8dh5\" (UID: \"825c4047-d1f9-4272-a188-f4b57843b902\") " pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f14cf9e9-32cf-4ce9-9761-16bed40d8a5e-serviceca\") pod \"node-ca-vf9gg\" (UID: \"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e\") " pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041777    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-daemon-config\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.042044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-var-lib-openvswitch\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041804    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ad7bbb9-a8d6-40d9-a962-3ce602311838-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-run-openvswitch\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-device-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/33d7b0af-08cf-4fa6-9f39-e461402d71f1-dbus\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041876    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041889    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-device-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/642babd4-24ba-47d0-899b-e8345129d52a-hosts-file\") pod \"node-resolver-xgs2m\" (UID: \"642babd4-24ba-47d0-899b-e8345129d52a\") " pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/825c4047-d1f9-4272-a188-f4b57843b902-iptables-alerter-script\") pod \"iptables-alerter-q8dh5\" (UID: \"825c4047-d1f9-4272-a188-f4b57843b902\") " pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.041989    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.041996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6m96\" (UniqueName: \"kubernetes.io/projected/a324c808-39af-4084-b96b-628e7899bbb2-kube-api-access-q6m96\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042004    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/33d7b0af-08cf-4fa6-9f39-e461402d71f1-dbus\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-os-release\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.042050    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret podName:33d7b0af-08cf-4fa6-9f39-e461402d71f1 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:43.542029329 +0000 UTC m=+3.026589923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret") pod "global-pull-secret-syncer-9k7ms" (UID: "33d7b0af-08cf-4fa6-9f39-e461402d71f1") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042075    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-os-release\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f14cf9e9-32cf-4ce9-9761-16bed40d8a5e-host\") pod \"node-ca-vf9gg\" (UID: \"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e\") " pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:43.042893 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042106    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f14cf9e9-32cf-4ce9-9761-16bed40d8a5e-host\") pod \"node-ca-vf9gg\" (UID: \"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e\") " pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqrpd\" (UniqueName: \"kubernetes.io/projected/4c568e24-a209-468a-b73f-7d8a0666559f-kube-api-access-cqrpd\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f14cf9e9-32cf-4ce9-9761-16bed40d8a5e-serviceca\") pod \"node-ca-vf9gg\" (UID: \"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e\") " pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-modprobe-d\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-var-lib-kubelet\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ad7bbb9-a8d6-40d9-a962-3ce602311838-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zqhg\" (UniqueName: \"kubernetes.io/projected/d71a09df-224a-475a-9c69-4fe776c7cdb3-kube-api-access-5zqhg\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-modprobe-d\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042235    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/825c4047-d1f9-4272-a188-f4b57843b902-host-slash\") pod \"iptables-alerter-q8dh5\" (UID: \"825c4047-d1f9-4272-a188-f4b57843b902\") " pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-var-lib-kubelet\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-host\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-system-cni-dir\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-etc-kubernetes\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042362    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/825c4047-d1f9-4272-a188-f4b57843b902-host-slash\") pod \"iptables-alerter-q8dh5\" (UID: \"825c4047-d1f9-4272-a188-f4b57843b902\") " pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-host\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ad7bbb9-a8d6-40d9-a962-3ce602311838-cni-binary-copy\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042391    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-etc-kubernetes\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.043596 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-system-cni-dir\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042433    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-slash\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-cni-netd\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-system-cni-dir\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-sys-fs\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-kubernetes\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042534    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-system-cni-dir\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-cni-dir\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042565    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/825c4047-d1f9-4272-a188-f4b57843b902-iptables-alerter-script\") pod \"iptables-alerter-q8dh5\" (UID: \"825c4047-d1f9-4272-a188-f4b57843b902\") " pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042596    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-sys-fs\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de658cfe-d206-4123-9309-e2fe7309d4e5-cni-binary-copy\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-kubernetes\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042661    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-cni-dir\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-socket-dir-parent\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ad7bbb9-a8d6-40d9-a962-3ce602311838-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9852s\" (UniqueName: \"kubernetes.io/projected/5a167781-23cd-4fec-880a-76ccee6874c5-kube-api-access-9852s\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.044341 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3a39934-0ab7-466e-a958-7ffebde7bb5a-agent-certs\") pod \"konnectivity-agent-rfc4r\" (UID: \"d3a39934-0ab7-466e-a958-7ffebde7bb5a\") " pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042773    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-socket-dir-parent\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042780    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ad7bbb9-a8d6-40d9-a962-3ce602311838-cni-binary-copy\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-socket-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/33d7b0af-08cf-4fa6-9f39-e461402d71f1-kubelet-config\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-sysconfig\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-sysctl-d\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-cnibin\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042919    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-socket-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-var-lib-cni-multus\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042934    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-kubelet\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042971    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-sysconfig\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-sysctl-conf\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043006    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-run-systemd\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043017    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-cnibin\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.045113 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-node-log\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043030    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de658cfe-d206-4123-9309-e2fe7309d4e5-cni-binary-copy\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.042984    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/33d7b0af-08cf-4fa6-9f39-e461402d71f1-kubelet-config\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-host-var-lib-cni-multus\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043069    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-sysctl-d\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a167781-23cd-4fec-880a-76ccee6874c5-env-overrides\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdctk\" (UniqueName: \"kubernetes.io/projected/642babd4-24ba-47d0-899b-e8345129d52a-kube-api-access-cdctk\") pod \"node-resolver-xgs2m\" (UID: \"642babd4-24ba-47d0-899b-e8345129d52a\") " pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-registration-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-hostroot\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5k6\" (UniqueName: \"kubernetes.io/projected/de658cfe-d206-4123-9309-e2fe7309d4e5-kube-api-access-kt5k6\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043185    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-etc-sysctl-conf\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-cnibin\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043219    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043226    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-cnibin\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-lib-modules\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a324c808-39af-4084-b96b-628e7899bbb2-etc-tuned\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043303    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c568e24-a209-468a-b73f-7d8a0666559f-registration-dir\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.045857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a324c808-39af-4084-b96b-628e7899bbb2-tmp\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.046723 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043347    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-hostroot\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.046723 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043441    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ad7bbb9-a8d6-40d9-a962-3ce602311838-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.046723 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043540    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a324c808-39af-4084-b96b-628e7899bbb2-lib-modules\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.046723 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.043607    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 15:40:43.046723 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.044033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-conf-dir\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.046723 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.044117    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de658cfe-d206-4123-9309-e2fe7309d4e5-multus-conf-dir\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.047285 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.047262    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:40:43.047285 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.047288    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:40:43.047505 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.047301    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wpd66 for pod openshift-network-diagnostics/network-check-target-kr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:43.047505 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.047364    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66 podName:b03c5f74-db11-4405-93ae-cd1b77d40f4a nodeName:}" failed. No retries permitted until 2026-07-24 15:40:43.547346046 +0000 UTC m=+3.031906841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd66" (UniqueName: "kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66") pod "network-check-target-kr79g" (UID: "b03c5f74-db11-4405-93ae-cd1b77d40f4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:43.047732 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.047519    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a324c808-39af-4084-b96b-628e7899bbb2-etc-tuned\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.047732 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.047516    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a324c808-39af-4084-b96b-628e7899bbb2-tmp\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.050953 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.050923    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-466xx\" (UniqueName: \"kubernetes.io/projected/0ad7bbb9-a8d6-40d9-a962-3ce602311838-kube-api-access-466xx\") pod \"multus-additional-cni-plugins-wk55j\" (UID: \"0ad7bbb9-a8d6-40d9-a962-3ce602311838\") " pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.050953 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.050923    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzj8z\" (UniqueName: \"kubernetes.io/projected/f14cf9e9-32cf-4ce9-9761-16bed40d8a5e-kube-api-access-wzj8z\") pod \"node-ca-vf9gg\" (UID: \"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e\") " pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:43.052535 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.052506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7kw8\" (UniqueName: \"kubernetes.io/projected/825c4047-d1f9-4272-a188-f4b57843b902-kube-api-access-j7kw8\") pod \"iptables-alerter-q8dh5\" (UID: \"825c4047-d1f9-4272-a188-f4b57843b902\") " pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:43.053506 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.053480    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5k6\" (UniqueName: \"kubernetes.io/projected/de658cfe-d206-4123-9309-e2fe7309d4e5-kube-api-access-kt5k6\") pod \"multus-tdm2k\" (UID: \"de658cfe-d206-4123-9309-e2fe7309d4e5\") " pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.054025 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.053989    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqrpd\" (UniqueName: \"kubernetes.io/projected/4c568e24-a209-468a-b73f-7d8a0666559f-kube-api-access-cqrpd\") pod \"aws-ebs-csi-driver-node-4c8x2\" (UID: \"4c568e24-a209-468a-b73f-7d8a0666559f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.054194 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.054175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6m96\" (UniqueName: \"kubernetes.io/projected/a324c808-39af-4084-b96b-628e7899bbb2-kube-api-access-q6m96\") pod \"tuned-5hqlm\" (UID: \"a324c808-39af-4084-b96b-628e7899bbb2\") " pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.144863 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.144827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:43.145128 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.144870    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-slash\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145128 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.144895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-cni-netd\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145128 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.144943    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-cni-netd\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145128 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.144965    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-slash\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145128 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.144973    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:43.145128 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145042    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9852s\" (UniqueName: \"kubernetes.io/projected/5a167781-23cd-4fec-880a-76ccee6874c5-kube-api-access-9852s\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145128 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3a39934-0ab7-466e-a958-7ffebde7bb5a-agent-certs\") pod \"konnectivity-agent-rfc4r\" (UID: \"d3a39934-0ab7-466e-a958-7ffebde7bb5a\") " pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:40:43.145128 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.145099    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs podName:d71a09df-224a-475a-9c69-4fe776c7cdb3 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:43.645077062 +0000 UTC m=+3.129637642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs") pod "network-metrics-daemon-2wl2t" (UID: "d71a09df-224a-475a-9c69-4fe776c7cdb3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-kubelet\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-run-systemd\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-node-log\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a167781-23cd-4fec-880a-76ccee6874c5-env-overrides\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdctk\" (UniqueName: \"kubernetes.io/projected/642babd4-24ba-47d0-899b-e8345129d52a-kube-api-access-cdctk\") pod \"node-resolver-xgs2m\" (UID: \"642babd4-24ba-47d0-899b-e8345129d52a\") " pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145304    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-systemd-units\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-log-socket\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145341    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-run-systemd\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145386    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-node-log\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145446 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145388    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-cni-bin\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145466    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-systemd-units\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145590    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-kubelet\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-cni-bin\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-run-netns\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a167781-23cd-4fec-880a-76ccee6874c5-ovnkube-script-lib\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-etc-openvswitch\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a167781-23cd-4fec-880a-76ccee6874c5-ovnkube-config\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3a39934-0ab7-466e-a958-7ffebde7bb5a-konnectivity-ca\") pod \"konnectivity-agent-rfc4r\" (UID: \"d3a39934-0ab7-466e-a958-7ffebde7bb5a\") " pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145728    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-log-socket\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.145847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145767    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-run-netns\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145855    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.145997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a167781-23cd-4fec-880a-76ccee6874c5-ovn-node-metrics-cert\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642babd4-24ba-47d0-899b-e8345129d52a-tmp-dir\") pod \"node-resolver-xgs2m\" (UID: \"642babd4-24ba-47d0-899b-e8345129d52a\") " pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-run-ovn\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-var-lib-openvswitch\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146110    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a167781-23cd-4fec-880a-76ccee6874c5-ovnkube-script-lib\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-run-openvswitch\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146175    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/642babd4-24ba-47d0-899b-e8345129d52a-hosts-file\") pod \"node-resolver-xgs2m\" (UID: \"642babd4-24ba-47d0-899b-e8345129d52a\") " pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zqhg\" (UniqueName: \"kubernetes.io/projected/d71a09df-224a-475a-9c69-4fe776c7cdb3-kube-api-access-5zqhg\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146346    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-run-ovn\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.146356 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146359    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a167781-23cd-4fec-880a-76ccee6874c5-env-overrides\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.147079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146393    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.147079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146411    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-run-openvswitch\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.147079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-var-lib-openvswitch\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.147079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/642babd4-24ba-47d0-899b-e8345129d52a-hosts-file\") pod \"node-resolver-xgs2m\" (UID: \"642babd4-24ba-47d0-899b-e8345129d52a\") " pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.147079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a167781-23cd-4fec-880a-76ccee6874c5-etc-openvswitch\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.147079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642babd4-24ba-47d0-899b-e8345129d52a-tmp-dir\") pod \"node-resolver-xgs2m\" (UID: \"642babd4-24ba-47d0-899b-e8345129d52a\") " pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.147079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a167781-23cd-4fec-880a-76ccee6874c5-ovnkube-config\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.147079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.146870    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3a39934-0ab7-466e-a958-7ffebde7bb5a-konnectivity-ca\") pod \"konnectivity-agent-rfc4r\" (UID: \"d3a39934-0ab7-466e-a958-7ffebde7bb5a\") " pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:40:43.148326 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.148142    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3a39934-0ab7-466e-a958-7ffebde7bb5a-agent-certs\") pod \"konnectivity-agent-rfc4r\" (UID: \"d3a39934-0ab7-466e-a958-7ffebde7bb5a\") " pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:40:43.149305 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.149267    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a167781-23cd-4fec-880a-76ccee6874c5-ovn-node-metrics-cert\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.154608 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.154565    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdctk\" (UniqueName: \"kubernetes.io/projected/642babd4-24ba-47d0-899b-e8345129d52a-kube-api-access-cdctk\") pod \"node-resolver-xgs2m\" (UID: \"642babd4-24ba-47d0-899b-e8345129d52a\") " pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.156066 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.156041    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zqhg\" (UniqueName: \"kubernetes.io/projected/d71a09df-224a-475a-9c69-4fe776c7cdb3-kube-api-access-5zqhg\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:43.157000 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.156977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9852s\" (UniqueName: \"kubernetes.io/projected/5a167781-23cd-4fec-880a-76ccee6874c5-kube-api-access-9852s\") pod \"ovnkube-node-mf4lp\" (UID: \"5a167781-23cd-4fec-880a-76ccee6874c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.171055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.171031    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 15:40:43.223141 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.223100    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q8dh5"
Jul 24 15:40:43.231008 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.230977    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vf9gg"
Jul 24 15:40:43.240932 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.240904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2"
Jul 24 15:40:43.247165 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.246954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5hqlm"
Jul 24 15:40:43.256779 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.256749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tdm2k"
Jul 24 15:40:43.265681 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.265654    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wk55j"
Jul 24 15:40:43.273469 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.273438    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:40:43.280354 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.280328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:40:43.287452 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.287435    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgs2m"
Jul 24 15:40:43.549368 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.549295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:43.549368 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.549335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:43.549551 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.549446    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:43.549551 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.549458    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:40:43.549551 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.549477    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:40:43.549551 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.549487    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wpd66 for pod openshift-network-diagnostics/network-check-target-kr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:43.549551 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.549512    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret podName:33d7b0af-08cf-4fa6-9f39-e461402d71f1 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:44.549496389 +0000 UTC m=+4.034056964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret") pod "global-pull-secret-syncer-9k7ms" (UID: "33d7b0af-08cf-4fa6-9f39-e461402d71f1") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:43.549551 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.549535    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66 podName:b03c5f74-db11-4405-93ae-cd1b77d40f4a nodeName:}" failed. No retries permitted until 2026-07-24 15:40:44.549518846 +0000 UTC m=+4.034079424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd66" (UniqueName: "kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66") pod "network-check-target-kr79g" (UID: "b03c5f74-db11-4405-93ae-cd1b77d40f4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:43.650607 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.650561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:43.650784 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.650729    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:43.650852 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:43.650795    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs podName:d71a09df-224a-475a-9c69-4fe776c7cdb3 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:44.650775437 +0000 UTC m=+4.135336013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs") pod "network-metrics-daemon-2wl2t" (UID: "d71a09df-224a-475a-9c69-4fe776c7cdb3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:43.955144 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:43.955005    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde658cfe_d206_4123_9309_e2fe7309d4e5.slice/crio-9c748c1dae15fa936376349009c2423efac9d97631d9d7b554a77492defb2000 WatchSource:0}: Error finding container 9c748c1dae15fa936376349009c2423efac9d97631d9d7b554a77492defb2000: Status 404 returned error can't find the container with id 9c748c1dae15fa936376349009c2423efac9d97631d9d7b554a77492defb2000
Jul 24 15:40:43.956905 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:43.956587    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad7bbb9_a8d6_40d9_a962_3ce602311838.slice/crio-5be1725a05572c4ca1eb56accb52c54fdeff0dd34602744b3af4108fa9d40e4a WatchSource:0}: Error finding container 5be1725a05572c4ca1eb56accb52c54fdeff0dd34602744b3af4108fa9d40e4a: Status 404 returned error can't find the container with id 5be1725a05572c4ca1eb56accb52c54fdeff0dd34602744b3af4108fa9d40e4a
Jul 24 15:40:43.960503 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:43.960414    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a167781_23cd_4fec_880a_76ccee6874c5.slice/crio-e8a6c9d452ef3fe766a1c4eaeec17931596810dbc75eb2f80fb714305c009cad WatchSource:0}: Error finding container e8a6c9d452ef3fe766a1c4eaeec17931596810dbc75eb2f80fb714305c009cad: Status 404 returned error can't find the container with id e8a6c9d452ef3fe766a1c4eaeec17931596810dbc75eb2f80fb714305c009cad
Jul 24 15:40:43.962572 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:43.962552    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825c4047_d1f9_4272_a188_f4b57843b902.slice/crio-b6e6f666ec9c7616f84217a78754672e63cb8a6a8feac05e1f7802311405f181 WatchSource:0}: Error finding container b6e6f666ec9c7616f84217a78754672e63cb8a6a8feac05e1f7802311405f181: Status 404 returned error can't find the container with id b6e6f666ec9c7616f84217a78754672e63cb8a6a8feac05e1f7802311405f181
Jul 24 15:40:43.963991 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:43.963725    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14cf9e9_32cf_4ce9_9761_16bed40d8a5e.slice/crio-1f3b2dee2286d3f289e18d51388e12d029b5cfb241df1ace311766130a1f33cd WatchSource:0}: Error finding container 1f3b2dee2286d3f289e18d51388e12d029b5cfb241df1ace311766130a1f33cd: Status 404 returned error can't find the container with id 1f3b2dee2286d3f289e18d51388e12d029b5cfb241df1ace311766130a1f33cd
Jul 24 15:40:43.963991 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:43.963963    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642babd4_24ba_47d0_899b_e8345129d52a.slice/crio-2b709190d1b7e69fda6e940a80972a0582710ea21a19ca33b6fd4774a131cd76 WatchSource:0}: Error finding container 2b709190d1b7e69fda6e940a80972a0582710ea21a19ca33b6fd4774a131cd76: Status 404 returned error can't find the container with id 2b709190d1b7e69fda6e940a80972a0582710ea21a19ca33b6fd4774a131cd76
Jul 24 15:40:43.964764 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:43.964744    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a39934_0ab7_466e_a958_7ffebde7bb5a.slice/crio-94ec1258905c6033f923534fd55389564654c9272ab90bc6a7669456ac39bfd5 WatchSource:0}: Error finding container 94ec1258905c6033f923534fd55389564654c9272ab90bc6a7669456ac39bfd5: Status 404 returned error can't find the container with id 94ec1258905c6033f923534fd55389564654c9272ab90bc6a7669456ac39bfd5
Jul 24 15:40:43.965968 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:40:43.965947    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c568e24_a209_468a_b73f_7d8a0666559f.slice/crio-b456d29d5a93730cb9967d433fee5d22cfdaae7aecccc47b9eea4ce4c3f37b11 WatchSource:0}: Error finding container b456d29d5a93730cb9967d433fee5d22cfdaae7aecccc47b9eea4ce4c3f37b11: Status 404 returned error can't find the container with id b456d29d5a93730cb9967d433fee5d22cfdaae7aecccc47b9eea4ce4c3f37b11
Jul 24 15:40:43.992408 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.992376    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:35:42 +0000 UTC" deadline="2028-04-22 13:32:36.049705504 +0000 UTC"
Jul 24 15:40:43.992408 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:43.992407    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15309h51m52.057302946s"
Jul 24 15:40:44.010774 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.010748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5hqlm" event={"ID":"a324c808-39af-4084-b96b-628e7899bbb2","Type":"ContainerStarted","Data":"3de0051132bd7c9c017e01a655a9b3c4dbfcbed7787d42d7c0f9cda51cf37901"}
Jul 24 15:40:44.011825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.011722    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2" event={"ID":"4c568e24-a209-468a-b73f-7d8a0666559f","Type":"ContainerStarted","Data":"b456d29d5a93730cb9967d433fee5d22cfdaae7aecccc47b9eea4ce4c3f37b11"}
Jul 24 15:40:44.013566 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.013548    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerStarted","Data":"e8a6c9d452ef3fe766a1c4eaeec17931596810dbc75eb2f80fb714305c009cad"}
Jul 24 15:40:44.014628 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.014610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk55j" event={"ID":"0ad7bbb9-a8d6-40d9-a962-3ce602311838","Type":"ContainerStarted","Data":"5be1725a05572c4ca1eb56accb52c54fdeff0dd34602744b3af4108fa9d40e4a"}
Jul 24 15:40:44.019281 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.019232    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tdm2k" event={"ID":"de658cfe-d206-4123-9309-e2fe7309d4e5","Type":"ContainerStarted","Data":"9c748c1dae15fa936376349009c2423efac9d97631d9d7b554a77492defb2000"}
Jul 24 15:40:44.022976 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.022955    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vf9gg" event={"ID":"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e","Type":"ContainerStarted","Data":"1f3b2dee2286d3f289e18d51388e12d029b5cfb241df1ace311766130a1f33cd"}
Jul 24 15:40:44.024476 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.024459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rfc4r" event={"ID":"d3a39934-0ab7-466e-a958-7ffebde7bb5a","Type":"ContainerStarted","Data":"94ec1258905c6033f923534fd55389564654c9272ab90bc6a7669456ac39bfd5"}
Jul 24 15:40:44.025355 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.025332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgs2m" event={"ID":"642babd4-24ba-47d0-899b-e8345129d52a","Type":"ContainerStarted","Data":"2b709190d1b7e69fda6e940a80972a0582710ea21a19ca33b6fd4774a131cd76"}
Jul 24 15:40:44.026091 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.026074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q8dh5" event={"ID":"825c4047-d1f9-4272-a188-f4b57843b902","Type":"ContainerStarted","Data":"b6e6f666ec9c7616f84217a78754672e63cb8a6a8feac05e1f7802311405f181"}
Jul 24 15:40:44.557448 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.556799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:44.557448 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.556864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:44.557448 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:44.556984    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:44.557448 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:44.557071    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret podName:33d7b0af-08cf-4fa6-9f39-e461402d71f1 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:46.557052942 +0000 UTC m=+6.041613518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret") pod "global-pull-secret-syncer-9k7ms" (UID: "33d7b0af-08cf-4fa6-9f39-e461402d71f1") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:44.557448 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:44.557381    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:40:44.557448 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:44.557405    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:40:44.557448 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:44.557418    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wpd66 for pod openshift-network-diagnostics/network-check-target-kr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:44.557789 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:44.557483    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66 podName:b03c5f74-db11-4405-93ae-cd1b77d40f4a nodeName:}" failed. No retries permitted until 2026-07-24 15:40:46.557467756 +0000 UTC m=+6.042028336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd66" (UniqueName: "kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66") pod "network-check-target-kr79g" (UID: "b03c5f74-db11-4405-93ae-cd1b77d40f4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:44.657931 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.657887    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:44.658125 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:44.658106    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:44.658213 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:44.658175    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs podName:d71a09df-224a-475a-9c69-4fe776c7cdb3 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:46.658157198 +0000 UTC m=+6.142717780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs") pod "network-metrics-daemon-2wl2t" (UID: "d71a09df-224a-475a-9c69-4fe776c7cdb3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:44.993444 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.993319    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:35:42 +0000 UTC" deadline="2028-04-18 20:11:04.717051152 +0000 UTC"
Jul 24 15:40:44.993444 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:44.993362    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15220h30m19.723693554s"
Jul 24 15:40:45.010675 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:45.010434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:45.010675 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:45.010434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:45.010906 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:45.010708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:45.016449 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:45.016021    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:40:45.016449 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:45.016379    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:40:45.016449 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:45.016416    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:40:45.045140 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:45.044943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal" event={"ID":"6924941272464cf155cde721c0cf91a1","Type":"ContainerStarted","Data":"18e65167aa1c2ad6c0b9539cf137e0c493f005f76b5b21ccfe505bae42e2ba61"}
Jul 24 15:40:45.053271 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:45.053016    2576 generic.go:356] "Generic (PLEG): container finished" podID="43b6e5fc9d1a935acaab964e139534e4" containerID="5955684539b4288b6038ef16d5a24ff6c4c03fed824bab1b3dd51db352e9152c" exitCode=0
Jul 24 15:40:45.053271 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:45.053066    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal" event={"ID":"43b6e5fc9d1a935acaab964e139534e4","Type":"ContainerDied","Data":"5955684539b4288b6038ef16d5a24ff6c4c03fed824bab1b3dd51db352e9152c"}
Jul 24 15:40:45.060965 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:45.060714    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal" podStartSLOduration=3.060063291 podStartE2EDuration="3.060063291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:40:45.059100598 +0000 UTC m=+4.543661196" watchObservedRunningTime="2026-07-24 15:40:45.060063291 +0000 UTC m=+4.544623880"
Jul 24 15:40:46.090675 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:46.090639    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal" event={"ID":"43b6e5fc9d1a935acaab964e139534e4","Type":"ContainerStarted","Data":"cc3eaa93c22a80cbb2bd9c37dc418a3504c8ea8274897ce391376b348596dd88"}
Jul 24 15:40:46.574769 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:46.574681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:46.574769 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:46.574769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:46.574995 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:46.574914    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:40:46.574995 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:46.574932    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:40:46.574995 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:46.574945    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wpd66 for pod openshift-network-diagnostics/network-check-target-kr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:46.575235 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:46.575000    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66 podName:b03c5f74-db11-4405-93ae-cd1b77d40f4a nodeName:}" failed. No retries permitted until 2026-07-24 15:40:50.57498195 +0000 UTC m=+10.059542532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd66" (UniqueName: "kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66") pod "network-check-target-kr79g" (UID: "b03c5f74-db11-4405-93ae-cd1b77d40f4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:46.575462 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:46.575402    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:46.575462 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:46.575458    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret podName:33d7b0af-08cf-4fa6-9f39-e461402d71f1 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:50.575441065 +0000 UTC m=+10.060001647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret") pod "global-pull-secret-syncer-9k7ms" (UID: "33d7b0af-08cf-4fa6-9f39-e461402d71f1") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:46.675604 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:46.675561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:46.675796 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:46.675756    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:46.675854 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:46.675821    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs podName:d71a09df-224a-475a-9c69-4fe776c7cdb3 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:50.675800783 +0000 UTC m=+10.160361382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs") pod "network-metrics-daemon-2wl2t" (UID: "d71a09df-224a-475a-9c69-4fe776c7cdb3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:47.000427 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:47.000348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:47.000427 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:47.000348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:47.000640 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:47.000475    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:40:47.000640 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:47.000528    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:40:47.000744 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:47.000645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:47.000744 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:47.000733    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:40:48.999779 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:48.999748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:49.000215 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:48.999885    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:40:49.000334 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:49.000310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:49.000448 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:49.000418    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:40:49.000519 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:49.000485    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:49.000628 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:49.000574    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:40:50.612975 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:50.612943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:50.613473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:50.612993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:50.613473 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:50.613095    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:50.613473 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:50.613112    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:40:50.613473 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:50.613133    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:40:50.613473 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:50.613147    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wpd66 for pod openshift-network-diagnostics/network-check-target-kr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:50.613473 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:50.613148    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret podName:33d7b0af-08cf-4fa6-9f39-e461402d71f1 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:58.61313431 +0000 UTC m=+18.097694886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret") pod "global-pull-secret-syncer-9k7ms" (UID: "33d7b0af-08cf-4fa6-9f39-e461402d71f1") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:50.613473 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:50.613206    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66 podName:b03c5f74-db11-4405-93ae-cd1b77d40f4a nodeName:}" failed. No retries permitted until 2026-07-24 15:40:58.613189899 +0000 UTC m=+18.097750480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd66" (UniqueName: "kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66") pod "network-check-target-kr79g" (UID: "b03c5f74-db11-4405-93ae-cd1b77d40f4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:50.713310 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:50.713273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:50.713448 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:50.713410    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:50.713484 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:50.713475    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs podName:d71a09df-224a-475a-9c69-4fe776c7cdb3 nodeName:}" failed. No retries permitted until 2026-07-24 15:40:58.713456868 +0000 UTC m=+18.198017450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs") pod "network-metrics-daemon-2wl2t" (UID: "d71a09df-224a-475a-9c69-4fe776c7cdb3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:51.001255 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:51.000775    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:51.001255 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:51.000870    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:51.001255 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:51.000882    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:40:51.001255 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:51.000958    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:40:51.001255 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:51.000974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:51.001255 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:51.001085    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:40:53.000141 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:53.000070    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:53.000141 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:53.000117    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:53.000637 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:53.000256    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:40:53.000637 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:53.000321    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:40:53.000637 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:53.000381    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:53.000637 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:53.000460    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:40:54.999257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:54.999205    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:54.999814 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:54.999205    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:54.999814 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:54.999365    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:40:54.999814 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:54.999391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:54.999814 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:54.999492    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:40:54.999814 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:54.999582    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:40:57.000119 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:57.000076    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:57.000119 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:57.000093    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:57.000649 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:57.000187    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:40:57.000649 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:57.000227    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:57.000649 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:57.000334    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:40:57.000649 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:57.000402    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:40:58.675344 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:58.675322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:58.675637 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:58.675382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:58.675637 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.675471    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:58.675637 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.675484    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:40:58.675637 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.675498    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:40:58.675637 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.675510    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wpd66 for pod openshift-network-diagnostics/network-check-target-kr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:58.675637 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.675535    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret podName:33d7b0af-08cf-4fa6-9f39-e461402d71f1 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:14.675516791 +0000 UTC m=+34.160077387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret") pod "global-pull-secret-syncer-9k7ms" (UID: "33d7b0af-08cf-4fa6-9f39-e461402d71f1") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:40:58.675637 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.675550    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66 podName:b03c5f74-db11-4405-93ae-cd1b77d40f4a nodeName:}" failed. No retries permitted until 2026-07-24 15:41:14.67554094 +0000 UTC m=+34.160101517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd66" (UniqueName: "kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66") pod "network-check-target-kr79g" (UID: "b03c5f74-db11-4405-93ae-cd1b77d40f4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:40:58.776537 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:58.776454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:58.776676 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.776585    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:58.776676 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.776647    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs podName:d71a09df-224a-475a-9c69-4fe776c7cdb3 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:14.776632897 +0000 UTC m=+34.261193473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs") pod "network-metrics-daemon-2wl2t" (UID: "d71a09df-224a-475a-9c69-4fe776c7cdb3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:40:58.999548 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:58.999509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:40:58.999548 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:58.999533    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:40:58.999768 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:40:58.999509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:40:58.999768 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.999646    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:40:58.999768 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.999705    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:40:58.999886 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:40:58.999773    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:41:00.999824 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:00.999795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:01.000304 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:00.999904    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:41:01.000304 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:01.000275    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:01.000408 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:01.000362    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:41:01.000408 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:01.000404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:01.000509 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:01.000466    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:41:02.999966 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:02.999763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:03.000788 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:02.999781    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:03.000788 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:02.999814    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:03.000788 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:03.000202    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:41:03.000788 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:03.000651    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:41:03.000788 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:03.000734    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:41:03.131784 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.131748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vf9gg" event={"ID":"f14cf9e9-32cf-4ce9-9761-16bed40d8a5e","Type":"ContainerStarted","Data":"4c60d8f464e9eb6e4cdb1271b6ba01c9d0cd389d02a72a3ee9526897ef3bce51"}
Jul 24 15:41:03.133293 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.133267    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rfc4r" event={"ID":"d3a39934-0ab7-466e-a958-7ffebde7bb5a","Type":"ContainerStarted","Data":"2524be02af496e5839a6fffa562cfd8bef3e53e95349cdf8a046c97ef9f2b0a1"}
Jul 24 15:41:03.134599 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.134551    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgs2m" event={"ID":"642babd4-24ba-47d0-899b-e8345129d52a","Type":"ContainerStarted","Data":"4c5919add467aed40ea55399bef26fc20e7c33c0d67fb1c82ecd7e22cfb58df9"}
Jul 24 15:41:03.135934 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.135913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5hqlm" event={"ID":"a324c808-39af-4084-b96b-628e7899bbb2","Type":"ContainerStarted","Data":"604761f44dadbf29aef547d60d6612ae41c462cb82f2a16fbb8d8c76af56fa8c"}
Jul 24 15:41:03.137359 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.137341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2" event={"ID":"4c568e24-a209-468a-b73f-7d8a0666559f","Type":"ContainerStarted","Data":"2a92d5e2a37529801d11ff973b7d69504f59fa8360f442e7ad6afd3610882621"}
Jul 24 15:41:03.139169 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.139150    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:41:03.139513 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.139492    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a167781-23cd-4fec-880a-76ccee6874c5" containerID="1c87305bf793058844be6c6b8c24cdbf4e3166f787b4f3de34af6136616cd3c5" exitCode=1
Jul 24 15:41:03.139608 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.139551    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerStarted","Data":"942e2d311d4e08ea92ee5863e3cbd63fcbbe83357a3714f5202def5a162b48b4"}
Jul 24 15:41:03.139608 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.139604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerDied","Data":"1c87305bf793058844be6c6b8c24cdbf4e3166f787b4f3de34af6136616cd3c5"}
Jul 24 15:41:03.139724 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.139653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerStarted","Data":"58c1b39aaf6f664170147c11512d2054dbeff7bb642dd78cb3dc0586bc0d0d05"}
Jul 24 15:41:03.140990 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.140972    2576 generic.go:356] "Generic (PLEG): container finished" podID="0ad7bbb9-a8d6-40d9-a962-3ce602311838" containerID="76e2eeecf7c40db7991f43599a4ac3b6704ca00b6a3839a2aac3661b26b95e40" exitCode=0
Jul 24 15:41:03.141075 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.140997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk55j" event={"ID":"0ad7bbb9-a8d6-40d9-a962-3ce602311838","Type":"ContainerDied","Data":"76e2eeecf7c40db7991f43599a4ac3b6704ca00b6a3839a2aac3661b26b95e40"}
Jul 24 15:41:03.142900 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.142882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tdm2k" event={"ID":"de658cfe-d206-4123-9309-e2fe7309d4e5","Type":"ContainerStarted","Data":"80ebf604daf46b79211c675f62aa2cdd4e93150e9034ac0e6e2b56faf8913e69"}
Jul 24 15:41:03.145926 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.145884    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vf9gg" podStartSLOduration=3.7322918290000002 podStartE2EDuration="22.145868643s" totalImagesPullingTime="18.413576814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:03.145738917 +0000 UTC m=+22.630299512" watchObservedRunningTime="2026-07-24 15:41:03.145868643 +0000 UTC m=+22.630429243"
Jul 24 15:41:03.146010 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.145986    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal" podStartSLOduration=21.145979687 podStartE2EDuration="21.145979687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:40:46.106451744 +0000 UTC m=+5.591012342" watchObservedRunningTime="2026-07-24 15:41:03.145979687 +0000 UTC m=+22.630540280"
Jul 24 15:41:03.185378 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.185332    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tdm2k" podStartSLOduration=3.398083904 podStartE2EDuration="22.185317s" totalImagesPullingTime="18.787233096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:03.185107521 +0000 UTC m=+22.669668112" watchObservedRunningTime="2026-07-24 15:41:03.185317 +0000 UTC m=+22.669877598"
Jul 24 15:41:03.199503 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.199379    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xgs2m" podStartSLOduration=3.713342547 podStartE2EDuration="22.199364376s" totalImagesPullingTime="18.486021829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:03.199193387 +0000 UTC m=+22.683753985" watchObservedRunningTime="2026-07-24 15:41:03.199364376 +0000 UTC m=+22.683924974"
Jul 24 15:41:03.215349 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.215304    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5hqlm" podStartSLOduration=3.735581695 podStartE2EDuration="22.215285748s" totalImagesPullingTime="18.479704053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:03.215137093 +0000 UTC m=+22.699697691" watchObservedRunningTime="2026-07-24 15:41:03.215285748 +0000 UTC m=+22.699846350"
Jul 24 15:41:03.236175 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:03.236118    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rfc4r" podStartSLOduration=3.824997525 podStartE2EDuration="22.236097636s" totalImagesPullingTime="18.411100111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:03.235175603 +0000 UTC m=+22.719736201" watchObservedRunningTime="2026-07-24 15:41:03.236097636 +0000 UTC m=+22.720658237"
Jul 24 15:41:04.134899 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.134871    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 15:41:04.146485 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.146459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q8dh5" event={"ID":"825c4047-d1f9-4272-a188-f4b57843b902","Type":"ContainerStarted","Data":"9035b293801dd054aa22b4e5d9e951fce3a0bd083c03817fba647c72f8b8cfdf"}
Jul 24 15:41:04.148746 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.148721    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2" event={"ID":"4c568e24-a209-468a-b73f-7d8a0666559f","Type":"ContainerStarted","Data":"7261faa32e3473883e4cadf5b14dda2360f7bd91e0eac5097a9e8f49db95a072"}
Jul 24 15:41:04.151699 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.151682    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:41:04.152146 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.152125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerStarted","Data":"988ebf8b7510864f559fe5f56315526f7dcf6bf232094c8efa57fc3ca34170e4"}
Jul 24 15:41:04.152232 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.152200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerStarted","Data":"dc089c4087e6e9c2a931d694d6627a3fba82a0cc698791dcaf7bde712e529ec7"}
Jul 24 15:41:04.152317 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.152258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerStarted","Data":"aee865cdae9b33a62218d37f76a007f8f198dbf5e0afcd766e587ef59dfb4191"}
Jul 24 15:41:04.160795 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.160756    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q8dh5" podStartSLOduration=4.701107384 podStartE2EDuration="23.160742772s" totalImagesPullingTime="18.459635388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:04.160514967 +0000 UTC m=+23.645075575" watchObservedRunningTime="2026-07-24 15:41:04.160742772 +0000 UTC m=+23.645303351"
Jul 24 15:41:04.993160 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.993028    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T15:41:04.134889867Z","UUID":"44509c6f-8eed-4a6a-9077-1719fac4cacd","Handler":null,"Name":"","Endpoint":""}
Jul 24 15:41:04.994917 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.994895    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 15:41:04.995039 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:04.994935    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 15:41:05.000154 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:05.000135    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:05.000293 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:05.000162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:05.000293 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:05.000271    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:41:05.000406 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:05.000325    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:41:05.000406 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:05.000366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:05.000495 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:05.000438    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:41:05.156963 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:05.156930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2" event={"ID":"4c568e24-a209-468a-b73f-7d8a0666559f","Type":"ContainerStarted","Data":"40e81697a79813bbfc8c764f6503313c562848e2571af57cdbd53527a193cc60"}
Jul 24 15:41:05.174400 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:05.174355    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4c8x2" podStartSLOduration=3.495451197 podStartE2EDuration="24.174341101s" totalImagesPullingTime="20.678889904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:05.174146208 +0000 UTC m=+24.658706980" watchObservedRunningTime="2026-07-24 15:41:05.174341101 +0000 UTC m=+24.658901697"
Jul 24 15:41:06.105437 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:06.105402    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:41:06.105986 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:06.105966    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:41:06.162546 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:06.162502    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:41:06.163000 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:06.162913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerStarted","Data":"0e451c7221ba6b9785e5326a9686dfe1ef838dea6601ea825b84f021236b0b1c"}
Jul 24 15:41:06.999392 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:06.999186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:06.999608 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:06.999186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:06.999608 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:06.999493    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:41:06.999608 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:06.999536    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:41:06.999608 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:06.999192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:06.999826 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:06.999632    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:41:08.999937 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:08.999755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:09.000521 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:08.999755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:09.000521 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:09.000016    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:41:09.000521 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:08.999816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:09.000521 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:09.000106    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:41:09.000521 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:09.000187    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:41:09.173229 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:09.173204    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:41:09.173576 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:09.173554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerStarted","Data":"2ad047d9738522ee77b67fcace55eee86022c698d1b86b1421d1f85f1db4f1dd"}
Jul 24 15:41:09.173734 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:09.173716    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:41:09.173865 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:09.173852    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:41:09.174003 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:09.173988    2576 scope.go:117] "RemoveContainer" containerID="1c87305bf793058844be6c6b8c24cdbf4e3166f787b4f3de34af6136616cd3c5"
Jul 24 15:41:09.175590 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:09.175562    2576 generic.go:356] "Generic (PLEG): container finished" podID="0ad7bbb9-a8d6-40d9-a962-3ce602311838" containerID="d24e6cb08a1715ffef7b6069634d73872bfad8b29f0313f572be21506ea89606" exitCode=0
Jul 24 15:41:09.175844 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:09.175605    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk55j" event={"ID":"0ad7bbb9-a8d6-40d9-a962-3ce602311838","Type":"ContainerDied","Data":"d24e6cb08a1715ffef7b6069634d73872bfad8b29f0313f572be21506ea89606"}
Jul 24 15:41:09.192114 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:09.192095    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:41:10.180993 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.180968    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:41:10.181441 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.181339    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" event={"ID":"5a167781-23cd-4fec-880a-76ccee6874c5","Type":"ContainerStarted","Data":"ea27c02d419420bb73398e6e185ee48c5aaabffa5ce29358b9f77a69f5fb33d1"}
Jul 24 15:41:10.181583 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.181563    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:41:10.183810 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.183781    2576 generic.go:356] "Generic (PLEG): container finished" podID="0ad7bbb9-a8d6-40d9-a962-3ce602311838" containerID="4631daf59b32954f12fdc82ce90abf47ba85587220747f018b906e510df6168d" exitCode=0
Jul 24 15:41:10.183911 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.183829    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk55j" event={"ID":"0ad7bbb9-a8d6-40d9-a962-3ce602311838","Type":"ContainerDied","Data":"4631daf59b32954f12fdc82ce90abf47ba85587220747f018b906e510df6168d"}
Jul 24 15:41:10.196768 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.196750    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:41:10.231865 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.231824    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp" podStartSLOduration=10.721991468 podStartE2EDuration="29.231811882s" totalImagesPullingTime="18.509820414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:10.207756002 +0000 UTC m=+29.692316598" watchObservedRunningTime="2026-07-24 15:41:10.231811882 +0000 UTC m=+29.716372476"
Jul 24 15:41:10.290150 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.290123    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kr79g"]
Jul 24 15:41:10.290265 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.290227    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:10.290357 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:10.290339    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:41:10.294397 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.294142    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9k7ms"]
Jul 24 15:41:10.294397 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.294234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:10.294397 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:10.294353    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:41:10.294831 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.294806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2wl2t"]
Jul 24 15:41:10.294953 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.294926    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:10.295047 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:10.295023    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:41:10.322400 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.322377    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:41:10.322499 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.322485    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 15:41:10.322989 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:10.322969    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rfc4r"
Jul 24 15:41:11.188312 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:11.188278    2576 generic.go:356] "Generic (PLEG): container finished" podID="0ad7bbb9-a8d6-40d9-a962-3ce602311838" containerID="2fc8bee0ed3f9de1c0a40e3f0b8948a9a84963e98f56fead7494a09c19383009" exitCode=0
Jul 24 15:41:11.188670 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:11.188350    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk55j" event={"ID":"0ad7bbb9-a8d6-40d9-a962-3ce602311838","Type":"ContainerDied","Data":"2fc8bee0ed3f9de1c0a40e3f0b8948a9a84963e98f56fead7494a09c19383009"}
Jul 24 15:41:11.999862 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:11.999773    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:11.999862 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:11.999800    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:12.000072 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:11.999774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:12.000072 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:11.999895    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:41:12.000072 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:12.000014    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:41:12.000168 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:12.000095    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:41:13.999764 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:13.999600    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:14.000206 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:13.999602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:14.000206 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:13.999836    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wl2t" podUID="d71a09df-224a-475a-9c69-4fe776c7cdb3"
Jul 24 15:41:14.000206 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:13.999602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:14.000206 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:13.999904    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr79g" podUID="b03c5f74-db11-4405-93ae-cd1b77d40f4a"
Jul 24 15:41:14.000206 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:13.999990    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9k7ms" podUID="33d7b0af-08cf-4fa6-9f39-e461402d71f1"
Jul 24 15:41:14.695092 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:14.695055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:14.695380 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:14.695125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:14.695380 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:14.695225    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 15:41:14.695380 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:14.695224    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 15:41:14.695380 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:14.695268    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 15:41:14.695380 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:14.695281    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wpd66 for pod openshift-network-diagnostics/network-check-target-kr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:41:14.695380 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:14.695301    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret podName:33d7b0af-08cf-4fa6-9f39-e461402d71f1 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:46.695286474 +0000 UTC m=+66.179847054 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret") pod "global-pull-secret-syncer-9k7ms" (UID: "33d7b0af-08cf-4fa6-9f39-e461402d71f1") : object "kube-system"/"original-pull-secret" not registered
Jul 24 15:41:14.695380 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:14.695326    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66 podName:b03c5f74-db11-4405-93ae-cd1b77d40f4a nodeName:}" failed. No retries permitted until 2026-07-24 15:41:46.695314287 +0000 UTC m=+66.179874863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpd66" (UniqueName: "kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66") pod "network-check-target-kr79g" (UID: "b03c5f74-db11-4405-93ae-cd1b77d40f4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 15:41:14.795641 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:14.795570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:14.795804 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:14.795725    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:41:14.795804 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:14.795790    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs podName:d71a09df-224a-475a-9c69-4fe776c7cdb3 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:46.795774771 +0000 UTC m=+66.280335349 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs") pod "network-metrics-daemon-2wl2t" (UID: "d71a09df-224a-475a-9c69-4fe776c7cdb3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 15:41:15.782193 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.782164    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeReady"
Jul 24 15:41:15.782762 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.782330    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 15:41:15.820318 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.820290    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79d4b6fdd4-cst5j"]
Jul 24 15:41:15.821973 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.821949    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:15.824499 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.824478    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 15:41:15.824597 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.824489    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 15:41:15.825682 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.825649    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wlmxz\""
Jul 24 15:41:15.825682 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.825658    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 15:41:15.836591 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.836568    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 15:41:15.838334 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.838315    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x5l4p"]
Jul 24 15:41:15.845467 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.845444    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tf88h"]
Jul 24 15:41:15.845682 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.845668    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5l4p"
Jul 24 15:41:15.847503 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.847485    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79d4b6fdd4-cst5j"]
Jul 24 15:41:15.847601 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.847586    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:15.848188 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.848154    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 15:41:15.848299 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.848190    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m8m7j\""
Jul 24 15:41:15.848299 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.848276    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 15:41:15.848299 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.848190    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 15:41:15.850387 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.850368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5l4p"]
Jul 24 15:41:15.850582 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.850564    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dz9d4\""
Jul 24 15:41:15.850662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.850620    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 15:41:15.850835 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.850815    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 15:41:15.855374 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.855356    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tf88h"]
Jul 24 15:41:15.902578 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.902557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxl7\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-kube-api-access-4wxl7\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:15.902688 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.902593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-trusted-ca\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:15.902688 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.902624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-bound-sa-token\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:15.902688 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.902666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d54a2a-55ae-4f11-99db-3f98a78db125-ca-trust-extracted\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:15.902804 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.902696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-image-registry-private-configuration\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:15.902804 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.902719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-certificates\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:15.902804 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.902774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:15.902906 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.902804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-installation-pull-secrets\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:15.982278 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.982207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rfc4r_d3a39934-0ab7-466e-a958-7ffebde7bb5a/konnectivity-agent/0.log" limit=1024
Jul 24 15:41:15.999584 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.999561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:15.999686 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.999667    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:15.999741 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:15.999704    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:16.003590 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-trusted-ca\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.003662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66b549b1-da81-4293-95c9-a7524f30bc40-config-volume\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.003662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003629    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nh5k9\""
Jul 24 15:41:16.003662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b549b1-da81-4293-95c9-a7524f30bc40-tmp-dir\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.003812 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003664    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 15:41:16.003812 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003677    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 15:41:16.003812 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-bound-sa-token\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.003812 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgxzk\" (UniqueName: \"kubernetes.io/projected/66b549b1-da81-4293-95c9-a7524f30bc40-kube-api-access-xgxzk\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.003812 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003784    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d54a2a-55ae-4f11-99db-3f98a78db125-ca-trust-extracted\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.003812 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003618    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hvmsc\""
Jul 24 15:41:16.004089 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-image-registry-private-configuration\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.004089 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-certificates\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.004089 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptn84\" (UniqueName: \"kubernetes.io/projected/76a699aa-4df7-4e65-bc5b-a87ab2c316e4-kube-api-access-ptn84\") pod \"ingress-canary-x5l4p\" (UID: \"76a699aa-4df7-4e65-bc5b-a87ab2c316e4\") " pod="openshift-ingress-canary/ingress-canary-x5l4p"
Jul 24 15:41:16.004089 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.004089 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-installation-pull-secrets\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.004089 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.004051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76a699aa-4df7-4e65-bc5b-a87ab2c316e4-cert\") pod \"ingress-canary-x5l4p\" (UID: \"76a699aa-4df7-4e65-bc5b-a87ab2c316e4\") " pod="openshift-ingress-canary/ingress-canary-x5l4p"
Jul 24 15:41:16.004426 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.003618    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 15:41:16.004426 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.004139    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d54a2a-55ae-4f11-99db-3f98a78db125-ca-trust-extracted\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.004426 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.004136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxl7\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-kube-api-access-4wxl7\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.004426 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.004222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.004426 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.004374    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 15:41:16.004426 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.004387    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d4b6fdd4-cst5j: secret "image-registry-tls" not found
Jul 24 15:41:16.004772 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.004442    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls podName:f7d54a2a-55ae-4f11-99db-3f98a78db125 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:16.504423938 +0000 UTC m=+35.988984514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls") pod "image-registry-79d4b6fdd4-cst5j" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125") : secret "image-registry-tls" not found
Jul 24 15:41:16.004772 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.004671    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-trusted-ca\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.006048 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.005891    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 15:41:16.006213 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.006188    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-certificates\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.008639 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.008618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-image-registry-private-configuration\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.008731 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.008640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-installation-pull-secrets\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.017498 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.017476    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-bound-sa-token\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.018730 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.018711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxl7\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-kube-api-access-4wxl7\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.068167 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.068138    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rfc4r_d3a39934-0ab7-466e-a958-7ffebde7bb5a/konnectivity-agent/0.log" limit=1024
Jul 24 15:41:16.105332 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.105307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.105439 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.105336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66b549b1-da81-4293-95c9-a7524f30bc40-config-volume\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.105439 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.105352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b549b1-da81-4293-95c9-a7524f30bc40-tmp-dir\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.105542 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.105466    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:41:16.105542 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.105478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgxzk\" (UniqueName: \"kubernetes.io/projected/66b549b1-da81-4293-95c9-a7524f30bc40-kube-api-access-xgxzk\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.105542 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.105518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptn84\" (UniqueName: \"kubernetes.io/projected/76a699aa-4df7-4e65-bc5b-a87ab2c316e4-kube-api-access-ptn84\") pod \"ingress-canary-x5l4p\" (UID: \"76a699aa-4df7-4e65-bc5b-a87ab2c316e4\") " pod="openshift-ingress-canary/ingress-canary-x5l4p"
Jul 24 15:41:16.105686 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.105545    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls podName:66b549b1-da81-4293-95c9-a7524f30bc40 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:16.605523563 +0000 UTC m=+36.090084143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls") pod "dns-default-tf88h" (UID: "66b549b1-da81-4293-95c9-a7524f30bc40") : secret "dns-default-metrics-tls" not found
Jul 24 15:41:16.105686 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.105641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76a699aa-4df7-4e65-bc5b-a87ab2c316e4-cert\") pod \"ingress-canary-x5l4p\" (UID: \"76a699aa-4df7-4e65-bc5b-a87ab2c316e4\") " pod="openshift-ingress-canary/ingress-canary-x5l4p"
Jul 24 15:41:16.105769 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.105687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b549b1-da81-4293-95c9-a7524f30bc40-tmp-dir\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.105924 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.105904    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66b549b1-da81-4293-95c9-a7524f30bc40-config-volume\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.108105 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.108078    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76a699aa-4df7-4e65-bc5b-a87ab2c316e4-cert\") pod \"ingress-canary-x5l4p\" (UID: \"76a699aa-4df7-4e65-bc5b-a87ab2c316e4\") " pod="openshift-ingress-canary/ingress-canary-x5l4p"
Jul 24 15:41:16.114045 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.114023    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptn84\" (UniqueName: \"kubernetes.io/projected/76a699aa-4df7-4e65-bc5b-a87ab2c316e4-kube-api-access-ptn84\") pod \"ingress-canary-x5l4p\" (UID: \"76a699aa-4df7-4e65-bc5b-a87ab2c316e4\") " pod="openshift-ingress-canary/ingress-canary-x5l4p"
Jul 24 15:41:16.114146 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.114131    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgxzk\" (UniqueName: \"kubernetes.io/projected/66b549b1-da81-4293-95c9-a7524f30bc40-kube-api-access-xgxzk\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.157848 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.157820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5l4p"
Jul 24 15:41:16.203506 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.203468    2576 generic.go:356] "Generic (PLEG): container finished" podID="0ad7bbb9-a8d6-40d9-a962-3ce602311838" containerID="1ceffc292f3b857f6e5a07c5237d71f38d5dc78c7f31b21c2540ac2940752118" exitCode=0
Jul 24 15:41:16.203506 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.203499    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk55j" event={"ID":"0ad7bbb9-a8d6-40d9-a962-3ce602311838","Type":"ContainerDied","Data":"1ceffc292f3b857f6e5a07c5237d71f38d5dc78c7f31b21c2540ac2940752118"}
Jul 24 15:41:16.295392 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.295219    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5l4p"]
Jul 24 15:41:16.299483 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:16.299436    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a699aa_4df7_4e65_bc5b_a87ab2c316e4.slice/crio-83f072c75a2727e7ca28dead6630f23108a4b60dac5f7e12b444326b5a1aa983 WatchSource:0}: Error finding container 83f072c75a2727e7ca28dead6630f23108a4b60dac5f7e12b444326b5a1aa983: Status 404 returned error can't find the container with id 83f072c75a2727e7ca28dead6630f23108a4b60dac5f7e12b444326b5a1aa983
Jul 24 15:41:16.508526 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.508494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:16.508655 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.508637    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 15:41:16.508703 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.508658    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d4b6fdd4-cst5j: secret "image-registry-tls" not found
Jul 24 15:41:16.508738 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.508711    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls podName:f7d54a2a-55ae-4f11-99db-3f98a78db125 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:17.508695653 +0000 UTC m=+36.993256229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls") pod "image-registry-79d4b6fdd4-cst5j" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125") : secret "image-registry-tls" not found
Jul 24 15:41:16.609485 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:16.609397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:16.609600 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.609509    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:41:16.609600 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:16.609556    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls podName:66b549b1-da81-4293-95c9-a7524f30bc40 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:17.609543661 +0000 UTC m=+37.094104237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls") pod "dns-default-tf88h" (UID: "66b549b1-da81-4293-95c9-a7524f30bc40") : secret "dns-default-metrics-tls" not found
Jul 24 15:41:17.207500 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:17.207466    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5l4p" event={"ID":"76a699aa-4df7-4e65-bc5b-a87ab2c316e4","Type":"ContainerStarted","Data":"83f072c75a2727e7ca28dead6630f23108a4b60dac5f7e12b444326b5a1aa983"}
Jul 24 15:41:17.211055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:17.211024    2576 generic.go:356] "Generic (PLEG): container finished" podID="0ad7bbb9-a8d6-40d9-a962-3ce602311838" containerID="6903af99dd994eeffa84783b156ef61eb58db2cfb6d4a30d93059e75d9da6752" exitCode=0
Jul 24 15:41:17.211193 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:17.211064    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk55j" event={"ID":"0ad7bbb9-a8d6-40d9-a962-3ce602311838","Type":"ContainerDied","Data":"6903af99dd994eeffa84783b156ef61eb58db2cfb6d4a30d93059e75d9da6752"}
Jul 24 15:41:17.515762 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:17.515669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:17.515926 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:17.515841    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 15:41:17.515926 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:17.515864    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d4b6fdd4-cst5j: secret "image-registry-tls" not found
Jul 24 15:41:17.516019 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:17.515931    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls podName:f7d54a2a-55ae-4f11-99db-3f98a78db125 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:19.515909087 +0000 UTC m=+39.000469698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls") pod "image-registry-79d4b6fdd4-cst5j" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125") : secret "image-registry-tls" not found
Jul 24 15:41:17.616476 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:17.616439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:17.616658 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:17.616612    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:41:17.616725 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:17.616695    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls podName:66b549b1-da81-4293-95c9-a7524f30bc40 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:19.616674577 +0000 UTC m=+39.101235156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls") pod "dns-default-tf88h" (UID: "66b549b1-da81-4293-95c9-a7524f30bc40") : secret "dns-default-metrics-tls" not found
Jul 24 15:41:18.217626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:18.217596    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wk55j" event={"ID":"0ad7bbb9-a8d6-40d9-a962-3ce602311838","Type":"ContainerStarted","Data":"035474c733232383ec69367063fdc384ba0e12f772867a30cb8dd2c56461ccb1"}
Jul 24 15:41:18.240269 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:18.240201    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wk55j" podStartSLOduration=7.517123348 podStartE2EDuration="37.240179805s" totalImagesPullingTime="29.723056457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:18.238748017 +0000 UTC m=+37.723308618" watchObservedRunningTime="2026-07-24 15:41:18.240179805 +0000 UTC m=+37.724740403"
Jul 24 15:41:18.362489 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:18.362402    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rfc4r_d3a39934-0ab7-466e-a958-7ffebde7bb5a/konnectivity-agent/0.log" limit=1024
Jul 24 15:41:18.461149 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:18.461109    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rfc4r_d3a39934-0ab7-466e-a958-7ffebde7bb5a/konnectivity-agent/0.log" limit=1024
Jul 24 15:41:19.221822 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:19.221635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5l4p" event={"ID":"76a699aa-4df7-4e65-bc5b-a87ab2c316e4","Type":"ContainerStarted","Data":"ce402316e50caa20edcf155b306e8b410586b0e6d3e962b8cc7495f5a1b5d7e6"}
Jul 24 15:41:19.531871 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:19.531794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:19.532047 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:19.531946    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 15:41:19.532047 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:19.531966    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d4b6fdd4-cst5j: secret "image-registry-tls" not found
Jul 24 15:41:19.532047 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:19.532032    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls podName:f7d54a2a-55ae-4f11-99db-3f98a78db125 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:23.532016032 +0000 UTC m=+43.016576607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls") pod "image-registry-79d4b6fdd4-cst5j" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125") : secret "image-registry-tls" not found
Jul 24 15:41:19.632803 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:19.632766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:19.632970 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:19.632905    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:41:19.633032 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:19.632977    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls podName:66b549b1-da81-4293-95c9-a7524f30bc40 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:23.632959411 +0000 UTC m=+43.117519991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls") pod "dns-default-tf88h" (UID: "66b549b1-da81-4293-95c9-a7524f30bc40") : secret "dns-default-metrics-tls" not found
Jul 24 15:41:20.678530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:20.678483    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rfc4r_d3a39934-0ab7-466e-a958-7ffebde7bb5a/konnectivity-agent/0.log" limit=1024
Jul 24 15:41:23.562391 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:23.562353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:23.562756 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:23.562508    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 15:41:23.562756 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:23.562533    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d4b6fdd4-cst5j: secret "image-registry-tls" not found
Jul 24 15:41:23.562756 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:23.562591    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls podName:f7d54a2a-55ae-4f11-99db-3f98a78db125 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:31.56257303 +0000 UTC m=+51.047133608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls") pod "image-registry-79d4b6fdd4-cst5j" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125") : secret "image-registry-tls" not found
Jul 24 15:41:23.662998 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:23.662972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:23.663131 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:23.663073    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:41:23.663131 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:23.663121    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls podName:66b549b1-da81-4293-95c9-a7524f30bc40 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:31.663108566 +0000 UTC m=+51.147669141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls") pod "dns-default-tf88h" (UID: "66b549b1-da81-4293-95c9-a7524f30bc40") : secret "dns-default-metrics-tls" not found
Jul 24 15:41:29.402015 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.401960    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x5l4p" podStartSLOduration=11.865850951 podStartE2EDuration="14.401943429s" totalImagesPullingTime="2.536092478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:19.236563028 +0000 UTC m=+38.721123628" watchObservedRunningTime="2026-07-24 15:41:29.401943429 +0000 UTC m=+48.886504026"
Jul 24 15:41:29.402437 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.402165    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-9b8th"]
Jul 24 15:41:29.448780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.448752    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"]
Jul 24 15:41:29.448928 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.448881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.451691 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.451665    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 24 15:41:29.451691 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.451690    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:41:29.451924 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.451693    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 24 15:41:29.451924 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.451700    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mbf9b\""
Jul 24 15:41:29.452088 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.452072    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 24 15:41:29.458134 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.458118    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 24 15:41:29.462180 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.462166    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rk57t"]
Jul 24 15:41:29.462295 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.462283    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.465141 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.465120    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 24 15:41:29.465264 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.465126    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 24 15:41:29.465264 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.465163    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:41:29.465264 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.465126    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ztktc\""
Jul 24 15:41:29.465656 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.465642    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 24 15:41:29.480542 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.480520    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-798f8fd9f9-zr8tr"]
Jul 24 15:41:29.480647 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.480631    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.483042 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.483019    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 24 15:41:29.483329 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.483313    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 24 15:41:29.483444 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.483343    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 15:41:29.483444 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.483352    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 15:41:29.483618 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.483605    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xsk8b\""
Jul 24 15:41:29.489538 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.489522    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 24 15:41:29.500631 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.500613    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-9b8th"]
Jul 24 15:41:29.500715 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.500642    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"]
Jul 24 15:41:29.500715 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.500662    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rk57t"]
Jul 24 15:41:29.500715 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.500685    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-798f8fd9f9-zr8tr"]
Jul 24 15:41:29.500825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.500734    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.501516 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.501481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9580587-864e-44e1-8d91-33cc9aa83cc5-config\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.501801 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.501781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9580587-864e-44e1-8d91-33cc9aa83cc5-serving-cert\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.501989 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.501970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9580587-864e-44e1-8d91-33cc9aa83cc5-trusted-ca\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.502139 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.502122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xshc\" (UniqueName: \"kubernetes.io/projected/b9580587-864e-44e1-8d91-33cc9aa83cc5-kube-api-access-5xshc\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.503180 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.503158    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 24 15:41:29.503418 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.503395    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 24 15:41:29.503604 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.503586    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 24 15:41:29.503859 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.503835    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 24 15:41:29.503944 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.503649    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 15:41:29.504129 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.504108    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 15:41:29.504343 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.504324    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jm9fp\""
Jul 24 15:41:29.510365 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.510348    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr"]
Jul 24 15:41:29.529660 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.529639    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"]
Jul 24 15:41:29.529770 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.529752    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr"
Jul 24 15:41:29.532474 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.532455    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xsq8l\""
Jul 24 15:41:29.532577 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.532486    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 24 15:41:29.532577 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.532490    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:41:29.553773 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.553749    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"]
Jul 24 15:41:29.553891 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.553880    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:29.556629 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.556612    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 15:41:29.557907 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.557891    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 24 15:41:29.557998 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.557957    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pgts6\""
Jul 24 15:41:29.557998 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.557985    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 15:41:29.558098 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.558046    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 24 15:41:29.577950 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.577927    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv"]
Jul 24 15:41:29.578066 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.578051    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.580467 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.580451    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:41:29.580737 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.580718    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 24 15:41:29.580805 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.580792    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 24 15:41:29.581035 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.581016    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 24 15:41:29.581035 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.581031    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-m8zjr\""
Jul 24 15:41:29.602033 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602008    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr"]
Jul 24 15:41:29.602133 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602042    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv"]
Jul 24 15:41:29.602133 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602055    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"]
Jul 24 15:41:29.602133 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602073    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"]
Jul 24 15:41:29.602322 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv"
Jul 24 15:41:29.602860 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2124fe0-2969-4497-862e-7ea34b7e487c-service-ca-bundle\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.602942 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kthp9\" (UniqueName: \"kubernetes.io/projected/c2124fe0-2969-4497-862e-7ea34b7e487c-kube-api-access-kthp9\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.602942 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/147ab95a-ccc0-46b6-88f6-553b434f3268-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:29.602942 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.603044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.602985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2pm\" (UniqueName: \"kubernetes.io/projected/8814c4dc-52a8-4447-a645-adecc1b46d89-kube-api-access-8t2pm\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.603044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9580587-864e-44e1-8d91-33cc9aa83cc5-config\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.603044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c2124fe0-2969-4497-862e-7ea34b7e487c-snapshots\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.603185 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2124fe0-2969-4497-862e-7ea34b7e487c-serving-cert\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.603185 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwmt5\" (UniqueName: \"kubernetes.io/projected/8e977190-f12c-4a1a-acbc-f262f1b70229-kube-api-access-qwmt5\") pod \"network-check-source-7d88cb55d8-zx4vv\" (UID: \"8e977190-f12c-4a1a-acbc-f262f1b70229\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv"
Jul 24 15:41:29.603185 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b3bbffa-0d24-4689-82a2-dc659dd0647a-config\") pod \"service-ca-operator-5d6c75f84f-j7ffk\" (UID: \"2b3bbffa-0d24-4689-82a2-dc659dd0647a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.603185 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b3bbffa-0d24-4689-82a2-dc659dd0647a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-j7ffk\" (UID: \"2b3bbffa-0d24-4689-82a2-dc659dd0647a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.603185 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4znn\" (UniqueName: \"kubernetes.io/projected/c2d5ba91-0564-4242-aae2-1239c9666af1-kube-api-access-w4znn\") pod \"volume-data-source-validator-549c6d54cf-d99lr\" (UID: \"c2d5ba91-0564-4242-aae2-1239c9666af1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr"
Jul 24 15:41:29.603457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9580587-864e-44e1-8d91-33cc9aa83cc5-trusted-ca\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.603457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9c1f7cf-fbd5-4107-903d-a6538f15c309-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7\" (UID: \"a9c1f7cf-fbd5-4107-903d-a6538f15c309\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.603457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2124fe0-2969-4497-862e-7ea34b7e487c-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.603457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:29.603457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xshc\" (UniqueName: \"kubernetes.io/projected/b9580587-864e-44e1-8d91-33cc9aa83cc5-kube-api-access-5xshc\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.603457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-default-certificate\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.603756 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv8ql\" (UniqueName: \"kubernetes.io/projected/a9c1f7cf-fbd5-4107-903d-a6538f15c309-kube-api-access-hv8ql\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7\" (UID: \"a9c1f7cf-fbd5-4107-903d-a6538f15c309\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.603756 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf52g\" (UniqueName: \"kubernetes.io/projected/147ab95a-ccc0-46b6-88f6-553b434f3268-kube-api-access-jf52g\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:29.603756 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.603756 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9580587-864e-44e1-8d91-33cc9aa83cc5-serving-cert\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.603756 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2124fe0-2969-4497-862e-7ea34b7e487c-tmp\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.603756 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68n25\" (UniqueName: \"kubernetes.io/projected/2b3bbffa-0d24-4689-82a2-dc659dd0647a-kube-api-access-68n25\") pod \"service-ca-operator-5d6c75f84f-j7ffk\" (UID: \"2b3bbffa-0d24-4689-82a2-dc659dd0647a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.603756 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-stats-auth\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.603756 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9c1f7cf-fbd5-4107-903d-a6538f15c309-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7\" (UID: \"a9c1f7cf-fbd5-4107-903d-a6538f15c309\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.603756 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603749    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9580587-864e-44e1-8d91-33cc9aa83cc5-config\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.604023 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.603984    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9580587-864e-44e1-8d91-33cc9aa83cc5-trusted-ca\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.604910 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.604891    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zr972\""
Jul 24 15:41:29.605707 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.605688    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"]
Jul 24 15:41:29.608890 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.608868    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9580587-864e-44e1-8d91-33cc9aa83cc5-serving-cert\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.621540 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.621517    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xshc\" (UniqueName: \"kubernetes.io/projected/b9580587-864e-44e1-8d91-33cc9aa83cc5-kube-api-access-5xshc\") pod \"console-operator-584dcb6b54-9b8th\" (UID: \"b9580587-864e-44e1-8d91-33cc9aa83cc5\") " pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.622130 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.622110    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"]
Jul 24 15:41:29.622344 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.622326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:29.624697 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.624672    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2bj9d\""
Jul 24 15:41:29.624697 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.624679    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 24 15:41:29.624847 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.624772    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 24 15:41:29.642608 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.642589    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"]
Jul 24 15:41:29.642696 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.642621    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"]
Jul 24 15:41:29.642737 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.642709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:29.645193 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.645172    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 24 15:41:29.645193 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.645179    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:41:29.645611 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.645591    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9f6dk\""
Jul 24 15:41:29.645709 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.645599    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 24 15:41:29.704158 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2pm\" (UniqueName: \"kubernetes.io/projected/8814c4dc-52a8-4447-a645-adecc1b46d89-kube-api-access-8t2pm\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.704158 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c2124fe0-2969-4497-862e-7ea34b7e487c-snapshots\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.704158 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2124fe0-2969-4497-862e-7ea34b7e487c-serving-cert\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.704158 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwmt5\" (UniqueName: \"kubernetes.io/projected/8e977190-f12c-4a1a-acbc-f262f1b70229-kube-api-access-qwmt5\") pod \"network-check-source-7d88cb55d8-zx4vv\" (UID: \"8e977190-f12c-4a1a-acbc-f262f1b70229\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv"
Jul 24 15:41:29.704530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e25ae61d-8fd4-499c-9d04-a58302f9886a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:29.704530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b3bbffa-0d24-4689-82a2-dc659dd0647a-config\") pod \"service-ca-operator-5d6c75f84f-j7ffk\" (UID: \"2b3bbffa-0d24-4689-82a2-dc659dd0647a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.704530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b3bbffa-0d24-4689-82a2-dc659dd0647a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-j7ffk\" (UID: \"2b3bbffa-0d24-4689-82a2-dc659dd0647a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.704530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4znn\" (UniqueName: \"kubernetes.io/projected/c2d5ba91-0564-4242-aae2-1239c9666af1-kube-api-access-w4znn\") pod \"volume-data-source-validator-549c6d54cf-d99lr\" (UID: \"c2d5ba91-0564-4242-aae2-1239c9666af1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr"
Jul 24 15:41:29.704530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9c1f7cf-fbd5-4107-903d-a6538f15c309-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7\" (UID: \"a9c1f7cf-fbd5-4107-903d-a6538f15c309\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.704530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2124fe0-2969-4497-862e-7ea34b7e487c-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.704825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:29.704825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvtj5\" (UniqueName: \"kubernetes.io/projected/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-kube-api-access-gvtj5\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:29.704825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-default-certificate\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.704825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv8ql\" (UniqueName: \"kubernetes.io/projected/a9c1f7cf-fbd5-4107-903d-a6538f15c309-kube-api-access-hv8ql\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7\" (UID: \"a9c1f7cf-fbd5-4107-903d-a6538f15c309\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.704825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf52g\" (UniqueName: \"kubernetes.io/projected/147ab95a-ccc0-46b6-88f6-553b434f3268-kube-api-access-jf52g\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:29.704825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:29.704825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.704825 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2124fe0-2969-4497-862e-7ea34b7e487c-tmp\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.705236 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68n25\" (UniqueName: \"kubernetes.io/projected/2b3bbffa-0d24-4689-82a2-dc659dd0647a-kube-api-access-68n25\") pod \"service-ca-operator-5d6c75f84f-j7ffk\" (UID: \"2b3bbffa-0d24-4689-82a2-dc659dd0647a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.705236 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704841    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c2124fe0-2969-4497-862e-7ea34b7e487c-snapshots\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.705236 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-stats-auth\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.705236 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:29.704935    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:41:29.705236 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:29.705005    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:30.204986727 +0000 UTC m=+49.689547303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : secret "router-metrics-certs-default" not found
Jul 24 15:41:29.705236 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.705208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9c1f7cf-fbd5-4107-903d-a6538f15c309-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7\" (UID: \"a9c1f7cf-fbd5-4107-903d-a6538f15c309\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.705561 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.705269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:29.705561 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.705315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2124fe0-2969-4497-862e-7ea34b7e487c-service-ca-bundle\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.705561 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.705348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kthp9\" (UniqueName: \"kubernetes.io/projected/c2124fe0-2969-4497-862e-7ea34b7e487c-kube-api-access-kthp9\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.705561 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.705378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/147ab95a-ccc0-46b6-88f6-553b434f3268-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:29.705561 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.705377    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9c1f7cf-fbd5-4107-903d-a6538f15c309-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7\" (UID: \"a9c1f7cf-fbd5-4107-903d-a6538f15c309\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.705561 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.705425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.705868 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:29.705577    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:30.205558718 +0000 UTC m=+49.690119313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : configmap references non-existent config key: service-ca.crt
Jul 24 15:41:29.705868 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.704838    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b3bbffa-0d24-4689-82a2-dc659dd0647a-config\") pod \"service-ca-operator-5d6c75f84f-j7ffk\" (UID: \"2b3bbffa-0d24-4689-82a2-dc659dd0647a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.705868 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.705720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2124fe0-2969-4497-862e-7ea34b7e487c-tmp\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.705868 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:29.705811    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:29.705868 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:29.705868    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls podName:147ab95a-ccc0-46b6-88f6-553b434f3268 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:30.205852322 +0000 UTC m=+49.690412898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-r9c4b" (UID: "147ab95a-ccc0-46b6-88f6-553b434f3268") : secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:29.706403 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.706233    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/147ab95a-ccc0-46b6-88f6-553b434f3268-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:29.706678 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.706643    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2124fe0-2969-4497-862e-7ea34b7e487c-service-ca-bundle\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.707048 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.707029    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b3bbffa-0d24-4689-82a2-dc659dd0647a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-j7ffk\" (UID: \"2b3bbffa-0d24-4689-82a2-dc659dd0647a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.707525 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.707505    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2124fe0-2969-4497-862e-7ea34b7e487c-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.707619 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.707557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2124fe0-2969-4497-862e-7ea34b7e487c-serving-cert\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.707726 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.707708    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9c1f7cf-fbd5-4107-903d-a6538f15c309-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7\" (UID: \"a9c1f7cf-fbd5-4107-903d-a6538f15c309\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.708015 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.707996    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-stats-auth\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.708144 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.708125    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-default-certificate\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.714171 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.714143    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4znn\" (UniqueName: \"kubernetes.io/projected/c2d5ba91-0564-4242-aae2-1239c9666af1-kube-api-access-w4znn\") pod \"volume-data-source-validator-549c6d54cf-d99lr\" (UID: \"c2d5ba91-0564-4242-aae2-1239c9666af1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr"
Jul 24 15:41:29.714482 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.714459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwmt5\" (UniqueName: \"kubernetes.io/projected/8e977190-f12c-4a1a-acbc-f262f1b70229-kube-api-access-qwmt5\") pod \"network-check-source-7d88cb55d8-zx4vv\" (UID: \"8e977190-f12c-4a1a-acbc-f262f1b70229\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv"
Jul 24 15:41:29.714833 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.714806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv8ql\" (UniqueName: \"kubernetes.io/projected/a9c1f7cf-fbd5-4107-903d-a6538f15c309-kube-api-access-hv8ql\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7\" (UID: \"a9c1f7cf-fbd5-4107-903d-a6538f15c309\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.715396 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.715357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68n25\" (UniqueName: \"kubernetes.io/projected/2b3bbffa-0d24-4689-82a2-dc659dd0647a-kube-api-access-68n25\") pod \"service-ca-operator-5d6c75f84f-j7ffk\" (UID: \"2b3bbffa-0d24-4689-82a2-dc659dd0647a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.715488 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.715459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2pm\" (UniqueName: \"kubernetes.io/projected/8814c4dc-52a8-4447-a645-adecc1b46d89-kube-api-access-8t2pm\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:29.715534 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.715494    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf52g\" (UniqueName: \"kubernetes.io/projected/147ab95a-ccc0-46b6-88f6-553b434f3268-kube-api-access-jf52g\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:29.716160 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.716144    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kthp9\" (UniqueName: \"kubernetes.io/projected/c2124fe0-2969-4497-862e-7ea34b7e487c-kube-api-access-kthp9\") pod \"insights-operator-5885fdb759-rk57t\" (UID: \"c2124fe0-2969-4497-862e-7ea34b7e487c\") " pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.758031 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.758000    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:29.770128 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.770099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"
Jul 24 15:41:29.789298 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.789274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-rk57t"
Jul 24 15:41:29.806830 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.806796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:29.806916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.806879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:29.806966 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:29.806937    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 15:41:29.806966 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.806949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e25ae61d-8fd4-499c-9d04-a58302f9886a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:29.807028 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:29.807009    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 15:41:29.807066 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:29.807014    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls podName:48b1c8e8-08ed-4097-8c80-ef6cd01541ee nodeName:}" failed. No retries permitted until 2026-07-24 15:41:30.306993063 +0000 UTC m=+49.791553639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls") pod "cluster-samples-operator-6748489656-msxpl" (UID: "48b1c8e8-08ed-4097-8c80-ef6cd01541ee") : secret "samples-operator-tls" not found
Jul 24 15:41:29.807112 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:29.807089    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert podName:e25ae61d-8fd4-499c-9d04-a58302f9886a nodeName:}" failed. No retries permitted until 2026-07-24 15:41:30.307072369 +0000 UTC m=+49.791632958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-6vwr4" (UID: "e25ae61d-8fd4-499c-9d04-a58302f9886a") : secret "networking-console-plugin-cert" not found
Jul 24 15:41:29.807148 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.807137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtj5\" (UniqueName: \"kubernetes.io/projected/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-kube-api-access-gvtj5\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:29.807649 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.807627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e25ae61d-8fd4-499c-9d04-a58302f9886a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:29.816534 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.816504    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtj5\" (UniqueName: \"kubernetes.io/projected/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-kube-api-access-gvtj5\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:29.839898 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.839873    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr"
Jul 24 15:41:29.886229 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.886015    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"
Jul 24 15:41:29.911314 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.911011    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-9b8th"]
Jul 24 15:41:29.917651 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.916446    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv"
Jul 24 15:41:29.926222 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:29.926185    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9580587_864e_44e1_8d91_33cc9aa83cc5.slice/crio-8504c4823cda7fb4c6a6344d1c352714d8509928dda2af977f896ee9e2e42f8a WatchSource:0}: Error finding container 8504c4823cda7fb4c6a6344d1c352714d8509928dda2af977f896ee9e2e42f8a: Status 404 returned error can't find the container with id 8504c4823cda7fb4c6a6344d1c352714d8509928dda2af977f896ee9e2e42f8a
Jul 24 15:41:29.978563 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:29.977884    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk"]
Jul 24 15:41:30.018200 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.017740    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-rk57t"]
Jul 24 15:41:30.021950 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:30.021916    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2124fe0_2969_4497_862e_7ea34b7e487c.slice/crio-6226f4fc31f1a61186330e19bc1115051edb3057c65f38a9616bca0da60a8dca WatchSource:0}: Error finding container 6226f4fc31f1a61186330e19bc1115051edb3057c65f38a9616bca0da60a8dca: Status 404 returned error can't find the container with id 6226f4fc31f1a61186330e19bc1115051edb3057c65f38a9616bca0da60a8dca
Jul 24 15:41:30.037919 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.037884    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr"]
Jul 24 15:41:30.041309 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:30.041279    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d5ba91_0564_4242_aae2_1239c9666af1.slice/crio-86220af6e55145f4765011aee36e79fba640dfe45add3eb962ca8afd2d7287b3 WatchSource:0}: Error finding container 86220af6e55145f4765011aee36e79fba640dfe45add3eb962ca8afd2d7287b3: Status 404 returned error can't find the container with id 86220af6e55145f4765011aee36e79fba640dfe45add3eb962ca8afd2d7287b3
Jul 24 15:41:30.083755 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.083633    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7"]
Jul 24 15:41:30.086116 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:30.086085    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c1f7cf_fbd5_4107_903d_a6538f15c309.slice/crio-086fbe864f966893a4becc120abf7ddf197ccea5d52c48dbbe705103e6fb66d2 WatchSource:0}: Error finding container 086fbe864f966893a4becc120abf7ddf197ccea5d52c48dbbe705103e6fb66d2: Status 404 returned error can't find the container with id 086fbe864f966893a4becc120abf7ddf197ccea5d52c48dbbe705103e6fb66d2
Jul 24 15:41:30.095704 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.095681    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv"]
Jul 24 15:41:30.098527 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:30.098490    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e977190_f12c_4a1a_acbc_f262f1b70229.slice/crio-ccf8b4eab6bfc68bd47041033a840c8ececd9cacb2a3f68ba81cc75112caf228 WatchSource:0}: Error finding container ccf8b4eab6bfc68bd47041033a840c8ececd9cacb2a3f68ba81cc75112caf228: Status 404 returned error can't find the container with id ccf8b4eab6bfc68bd47041033a840c8ececd9cacb2a3f68ba81cc75112caf228
Jul 24 15:41:30.211066 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.210983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:30.211205 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.211078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:30.211205 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.211118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:30.211205 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:30.211128    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:30.211205 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:30.211183    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls podName:147ab95a-ccc0-46b6-88f6-553b434f3268 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:31.211168657 +0000 UTC m=+50.695729233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-r9c4b" (UID: "147ab95a-ccc0-46b6-88f6-553b434f3268") : secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:30.211377 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:30.211216    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:41:30.211377 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:30.211288    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:31.211272472 +0000 UTC m=+50.695833047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : configmap references non-existent config key: service-ca.crt
Jul 24 15:41:30.211377 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:30.211302    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:31.211296013 +0000 UTC m=+50.695856589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : secret "router-metrics-certs-default" not found
Jul 24 15:41:30.248607 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.248571    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7" event={"ID":"a9c1f7cf-fbd5-4107-903d-a6538f15c309","Type":"ContainerStarted","Data":"086fbe864f966893a4becc120abf7ddf197ccea5d52c48dbbe705103e6fb66d2"}
Jul 24 15:41:30.249589 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.249555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv" event={"ID":"8e977190-f12c-4a1a-acbc-f262f1b70229","Type":"ContainerStarted","Data":"ccf8b4eab6bfc68bd47041033a840c8ececd9cacb2a3f68ba81cc75112caf228"}
Jul 24 15:41:30.250571 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.250544    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" event={"ID":"b9580587-864e-44e1-8d91-33cc9aa83cc5","Type":"ContainerStarted","Data":"8504c4823cda7fb4c6a6344d1c352714d8509928dda2af977f896ee9e2e42f8a"}
Jul 24 15:41:30.251569 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.251546    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr" event={"ID":"c2d5ba91-0564-4242-aae2-1239c9666af1","Type":"ContainerStarted","Data":"86220af6e55145f4765011aee36e79fba640dfe45add3eb962ca8afd2d7287b3"}
Jul 24 15:41:30.252535 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.252519    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rk57t" event={"ID":"c2124fe0-2969-4497-862e-7ea34b7e487c","Type":"ContainerStarted","Data":"6226f4fc31f1a61186330e19bc1115051edb3057c65f38a9616bca0da60a8dca"}
Jul 24 15:41:30.253493 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.253470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk" event={"ID":"2b3bbffa-0d24-4689-82a2-dc659dd0647a","Type":"ContainerStarted","Data":"8fbc42a0d2119310fcc3ac78aaf1d50fa8d7b0c8f693b9b34d9a0b837f476993"}
Jul 24 15:41:30.312178 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.312141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:30.312363 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:30.312204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:30.312363 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:30.312334    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 15:41:30.312363 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:30.312353    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 15:41:30.312464 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:30.312400    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls podName:48b1c8e8-08ed-4097-8c80-ef6cd01541ee nodeName:}" failed. No retries permitted until 2026-07-24 15:41:31.312384184 +0000 UTC m=+50.796944760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls") pod "cluster-samples-operator-6748489656-msxpl" (UID: "48b1c8e8-08ed-4097-8c80-ef6cd01541ee") : secret "samples-operator-tls" not found
Jul 24 15:41:30.312464 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:30.312414    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert podName:e25ae61d-8fd4-499c-9d04-a58302f9886a nodeName:}" failed. No retries permitted until 2026-07-24 15:41:31.312408069 +0000 UTC m=+50.796968644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-6vwr4" (UID: "e25ae61d-8fd4-499c-9d04-a58302f9886a") : secret "networking-console-plugin-cert" not found
Jul 24 15:41:31.224279 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:31.224222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:31.224760 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:31.224363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:31.224760 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:31.224432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:31.224760 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.224600    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:33.22457825 +0000 UTC m=+52.709138856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : configmap references non-existent config key: service-ca.crt
Jul 24 15:41:31.225139 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.225033    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:31.225139 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.225092    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls podName:147ab95a-ccc0-46b6-88f6-553b434f3268 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:33.225076584 +0000 UTC m=+52.709637178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-r9c4b" (UID: "147ab95a-ccc0-46b6-88f6-553b434f3268") : secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:31.225363 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.225151    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:41:31.225363 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.225195    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:33.225182616 +0000 UTC m=+52.709743207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : secret "router-metrics-certs-default" not found
Jul 24 15:41:31.325230 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:31.325193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:31.325472 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:31.325401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:31.325639 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.325569    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 15:41:31.325639 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.325634    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls podName:48b1c8e8-08ed-4097-8c80-ef6cd01541ee nodeName:}" failed. No retries permitted until 2026-07-24 15:41:33.32561401 +0000 UTC m=+52.810174589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls") pod "cluster-samples-operator-6748489656-msxpl" (UID: "48b1c8e8-08ed-4097-8c80-ef6cd01541ee") : secret "samples-operator-tls" not found
Jul 24 15:41:31.325906 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.325707    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 15:41:31.325906 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.325743    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert podName:e25ae61d-8fd4-499c-9d04-a58302f9886a nodeName:}" failed. No retries permitted until 2026-07-24 15:41:33.325731184 +0000 UTC m=+52.810291763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-6vwr4" (UID: "e25ae61d-8fd4-499c-9d04-a58302f9886a") : secret "networking-console-plugin-cert" not found
Jul 24 15:41:31.628338 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:31.628257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:31.628607 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.628551    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 15:41:31.628607 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.628571    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d4b6fdd4-cst5j: secret "image-registry-tls" not found
Jul 24 15:41:31.628743 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.628628    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls podName:f7d54a2a-55ae-4f11-99db-3f98a78db125 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:47.628610111 +0000 UTC m=+67.113170708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls") pod "image-registry-79d4b6fdd4-cst5j" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125") : secret "image-registry-tls" not found
Jul 24 15:41:31.729840 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:31.729809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:31.730002 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.729933    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 15:41:31.730002 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:31.729998    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls podName:66b549b1-da81-4293-95c9-a7524f30bc40 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:47.729980253 +0000 UTC m=+67.214540832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls") pod "dns-default-tf88h" (UID: "66b549b1-da81-4293-95c9-a7524f30bc40") : secret "dns-default-metrics-tls" not found
Jul 24 15:41:33.245713 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:33.245677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:33.246080 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:33.245773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:33.246080 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:33.245818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:33.246080 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:33.245829    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:33.246080 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:33.245892    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls podName:147ab95a-ccc0-46b6-88f6-553b434f3268 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:37.245875793 +0000 UTC m=+56.730436373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-r9c4b" (UID: "147ab95a-ccc0-46b6-88f6-553b434f3268") : secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:33.246080 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:33.245925    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:41:33.246080 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:33.245951    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:37.245936459 +0000 UTC m=+56.730497055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : configmap references non-existent config key: service-ca.crt
Jul 24 15:41:33.246080 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:33.245973    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:37.245961475 +0000 UTC m=+56.730522077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : secret "router-metrics-certs-default" not found
Jul 24 15:41:33.346373 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:33.346341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:33.346607 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:33.346490    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 15:41:33.346607 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:33.346490    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:33.346607 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:33.346556    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert podName:e25ae61d-8fd4-499c-9d04-a58302f9886a nodeName:}" failed. No retries permitted until 2026-07-24 15:41:37.346540715 +0000 UTC m=+56.831101301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-6vwr4" (UID: "e25ae61d-8fd4-499c-9d04-a58302f9886a") : secret "networking-console-plugin-cert" not found
Jul 24 15:41:33.346607 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:33.346566    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 15:41:33.346744 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:33.346616    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls podName:48b1c8e8-08ed-4097-8c80-ef6cd01541ee nodeName:}" failed. No retries permitted until 2026-07-24 15:41:37.346604487 +0000 UTC m=+56.831165073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls") pod "cluster-samples-operator-6748489656-msxpl" (UID: "48b1c8e8-08ed-4097-8c80-ef6cd01541ee") : secret "samples-operator-tls" not found
Jul 24 15:41:37.277883 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.277587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7" event={"ID":"a9c1f7cf-fbd5-4107-903d-a6538f15c309","Type":"ContainerStarted","Data":"36202b48f2f10dffb9f1dc92635ffeeec62736349223b4394600ed373e6a7755"}
Jul 24 15:41:37.279567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.279538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv" event={"ID":"8e977190-f12c-4a1a-acbc-f262f1b70229","Type":"ContainerStarted","Data":"9ac88ca47a4a16b3cef91b0f422cdc528b774a7eb0b94c5542db551437b56d67"}
Jul 24 15:41:37.281408 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.281376    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/0.log"
Jul 24 15:41:37.281524 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.281413    2576 generic.go:356] "Generic (PLEG): container finished" podID="b9580587-864e-44e1-8d91-33cc9aa83cc5" containerID="87ed7893f0a72f914b3105aebdaf0433d87dac4022a9f9d0df69224421fb3e5a" exitCode=255
Jul 24 15:41:37.281668 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.281648    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" event={"ID":"b9580587-864e-44e1-8d91-33cc9aa83cc5","Type":"ContainerDied","Data":"87ed7893f0a72f914b3105aebdaf0433d87dac4022a9f9d0df69224421fb3e5a"}
Jul 24 15:41:37.281776 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.281727    2576 scope.go:117] "RemoveContainer" containerID="87ed7893f0a72f914b3105aebdaf0433d87dac4022a9f9d0df69224421fb3e5a"
Jul 24 15:41:37.283531 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.283510    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr" event={"ID":"c2d5ba91-0564-4242-aae2-1239c9666af1","Type":"ContainerStarted","Data":"135d2f477b12fa734be7e167b723414227546cf3e9306b1a8e5251f2e54169e1"}
Jul 24 15:41:37.285317 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.285215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:37.285317 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.285274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rk57t" event={"ID":"c2124fe0-2969-4497-862e-7ea34b7e487c","Type":"ContainerStarted","Data":"ca14bd51a3de24b9b7018fbe899aec427cd04f6c5367e8e1e8acd0f80f9a38cc"}
Jul 24 15:41:37.285484 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.285327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:37.285484 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.285381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:37.285612 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:37.285495    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:37.285612 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:37.285535    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:41:37.285612 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:37.285568    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls podName:147ab95a-ccc0-46b6-88f6-553b434f3268 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:45.285547456 +0000 UTC m=+64.770108037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-r9c4b" (UID: "147ab95a-ccc0-46b6-88f6-553b434f3268") : secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:37.285612 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:37.285591    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:45.285575075 +0000 UTC m=+64.770135654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : secret "router-metrics-certs-default" not found
Jul 24 15:41:37.285995 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:37.285651    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:45.285640794 +0000 UTC m=+64.770201375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : configmap references non-existent config key: service-ca.crt
Jul 24 15:41:37.286732 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.286712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk" event={"ID":"2b3bbffa-0d24-4689-82a2-dc659dd0647a","Type":"ContainerStarted","Data":"42eabc5a9320b1a0e0085a5649cd8331db4358a585f863f0a2d2c276cee26fb6"}
Jul 24 15:41:37.295398 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.295285    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7" podStartSLOduration=2.034553332 podStartE2EDuration="8.295269893s" totalImagesPullingTime="6.260716561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:37.295221281 +0000 UTC m=+56.779781880" watchObservedRunningTime="2026-07-24 15:41:37.295269893 +0000 UTC m=+56.779830489"
Jul 24 15:41:37.313312 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.313263    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-rk57t" podStartSLOduration=1.9889160110000002 podStartE2EDuration="8.313220553s" totalImagesPullingTime="6.324304542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:37.312599477 +0000 UTC m=+56.797160074" watchObservedRunningTime="2026-07-24 15:41:37.313220553 +0000 UTC m=+56.797781156"
Jul 24 15:41:37.348109 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.347716    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk" podStartSLOduration=1.9843144289999999 podStartE2EDuration="8.347695394s" totalImagesPullingTime="6.363380965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:37.346461979 +0000 UTC m=+56.831022592" watchObservedRunningTime="2026-07-24 15:41:37.347695394 +0000 UTC m=+56.832255993"
Jul 24 15:41:37.368827 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.368682    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zx4vv" podStartSLOduration=2.109607006 podStartE2EDuration="8.368662349s" totalImagesPullingTime="6.259055343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:37.36721038 +0000 UTC m=+56.851770979" watchObservedRunningTime="2026-07-24 15:41:37.368662349 +0000 UTC m=+56.853222947"
Jul 24 15:41:37.389697 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.387551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:37.389697 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.387686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:37.389697 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:37.388358    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d99lr" podStartSLOduration=2.514924095 podStartE2EDuration="8.388340377s" totalImagesPullingTime="5.873416282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:37.386753512 +0000 UTC m=+56.871314133" watchObservedRunningTime="2026-07-24 15:41:37.388340377 +0000 UTC m=+56.872900976"
Jul 24 15:41:37.389697 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:37.388909    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 15:41:37.389697 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:37.388961    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls podName:48b1c8e8-08ed-4097-8c80-ef6cd01541ee nodeName:}" failed. No retries permitted until 2026-07-24 15:41:45.388946243 +0000 UTC m=+64.873506835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls") pod "cluster-samples-operator-6748489656-msxpl" (UID: "48b1c8e8-08ed-4097-8c80-ef6cd01541ee") : secret "samples-operator-tls" not found
Jul 24 15:41:37.389697 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:37.389026    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 15:41:37.389697 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:37.389087    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert podName:e25ae61d-8fd4-499c-9d04-a58302f9886a nodeName:}" failed. No retries permitted until 2026-07-24 15:41:45.389071166 +0000 UTC m=+64.873631745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-6vwr4" (UID: "e25ae61d-8fd4-499c-9d04-a58302f9886a") : secret "networking-console-plugin-cert" not found
Jul 24 15:41:38.292450 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.292424    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/1.log"
Jul 24 15:41:38.292924 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.292837    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/0.log"
Jul 24 15:41:38.292924 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.292872    2576 generic.go:356] "Generic (PLEG): container finished" podID="b9580587-864e-44e1-8d91-33cc9aa83cc5" containerID="345a5b820ecb33d5383ed27e702cccc8d5d2a960bdcc7e1fbbf2c336a04febd9" exitCode=255
Jul 24 15:41:38.293028 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.292929    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" event={"ID":"b9580587-864e-44e1-8d91-33cc9aa83cc5","Type":"ContainerDied","Data":"345a5b820ecb33d5383ed27e702cccc8d5d2a960bdcc7e1fbbf2c336a04febd9"}
Jul 24 15:41:38.293028 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.292997    2576 scope.go:117] "RemoveContainer" containerID="87ed7893f0a72f914b3105aebdaf0433d87dac4022a9f9d0df69224421fb3e5a"
Jul 24 15:41:38.293209 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.293194    2576 scope.go:117] "RemoveContainer" containerID="345a5b820ecb33d5383ed27e702cccc8d5d2a960bdcc7e1fbbf2c336a04febd9"
Jul 24 15:41:38.293436 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:38.293407    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-9b8th_openshift-console-operator(b9580587-864e-44e1-8d91-33cc9aa83cc5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" podUID="b9580587-864e-44e1-8d91-33cc9aa83cc5"
Jul 24 15:41:38.421591 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.421551    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lx822"]
Jul 24 15:41:38.425641 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.425613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.428309 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.428273    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 15:41:38.428603 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.428582    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtzxv\""
Jul 24 15:41:38.434576 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.432623    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 15:41:38.438581 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.438555    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lx822"]
Jul 24 15:41:38.499719 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.499682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.499883 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.499735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d245c2e-8405-4b45-927b-cd0bcfbd5883-data-volume\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.499883 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.499804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppfv\" (UniqueName: \"kubernetes.io/projected/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-api-access-zppfv\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.499883 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.499841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d245c2e-8405-4b45-927b-cd0bcfbd5883-crio-socket\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.500099 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.499906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.601289 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.601171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppfv\" (UniqueName: \"kubernetes.io/projected/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-api-access-zppfv\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.601289 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.601236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d245c2e-8405-4b45-927b-cd0bcfbd5883-crio-socket\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.601289 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.601286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.601568 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.601421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.601568 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.601463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d245c2e-8405-4b45-927b-cd0bcfbd5883-data-volume\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.601568 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.601475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d245c2e-8405-4b45-927b-cd0bcfbd5883-crio-socket\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.601568 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:38.601492    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:38.601568 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:38.601551    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:41:38.601792 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:38.601580    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:39.101557473 +0000 UTC m=+58.586118050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:38.601792 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:38.601614    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:39.101601184 +0000 UTC m=+58.586161761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : configmap "kube-rbac-proxy" not found
Jul 24 15:41:38.601900 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.601832    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d245c2e-8405-4b45-927b-cd0bcfbd5883-data-volume\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:38.609717 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:38.609689    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppfv\" (UniqueName: \"kubernetes.io/projected/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-api-access-zppfv\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:39.106986 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:39.106942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:39.107168 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:39.107089    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:39.107168 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:39.107110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:39.107168 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:39.107162    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:40.107146292 +0000 UTC m=+59.591706868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:39.107319 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:39.107198    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:41:39.107319 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:39.107286    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:40.107269792 +0000 UTC m=+59.591830373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : configmap "kube-rbac-proxy" not found
Jul 24 15:41:39.299877 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:39.299848    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/1.log"
Jul 24 15:41:39.300381 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:39.300192    2576 scope.go:117] "RemoveContainer" containerID="345a5b820ecb33d5383ed27e702cccc8d5d2a960bdcc7e1fbbf2c336a04febd9"
Jul 24 15:41:39.300381 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:39.300368    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-9b8th_openshift-console-operator(b9580587-864e-44e1-8d91-33cc9aa83cc5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" podUID="b9580587-864e-44e1-8d91-33cc9aa83cc5"
Jul 24 15:41:39.758712 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:39.758672    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:39.758873 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:39.758746    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:40.116479 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:40.116372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:40.116643 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:40.116529    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:40.116643 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:40.116544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:40.116643 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:40.116590    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:42.116574433 +0000 UTC m=+61.601135009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:40.116643 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:40.116620    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:41:40.116807 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:40.116675    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:42.116658341 +0000 UTC m=+61.601218920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : configmap "kube-rbac-proxy" not found
Jul 24 15:41:40.234867 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:40.234839    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgs2m_642babd4-24ba-47d0-899b-e8345129d52a/dns-node-resolver/0.log"
Jul 24 15:41:40.303299 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:40.303271    2576 scope.go:117] "RemoveContainer" containerID="345a5b820ecb33d5383ed27e702cccc8d5d2a960bdcc7e1fbbf2c336a04febd9"
Jul 24 15:41:40.303772 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:40.303492    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-9b8th_openshift-console-operator(b9580587-864e-44e1-8d91-33cc9aa83cc5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" podUID="b9580587-864e-44e1-8d91-33cc9aa83cc5"
Jul 24 15:41:41.032680 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:41.032650    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vf9gg_f14cf9e9-32cf-4ce9-9761-16bed40d8a5e/node-ca/0.log"
Jul 24 15:41:41.832217 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:41.832176    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x5l4p_76a699aa-4df7-4e65-bc5b-a87ab2c316e4/serve-healthcheck-canary/0.log"
Jul 24 15:41:42.136313 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:42.136183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:42.136313 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:42.136292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:42.136499 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:42.136335    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:41:42.136499 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:42.136398    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:46.136383127 +0000 UTC m=+65.620943703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : configmap "kube-rbac-proxy" not found
Jul 24 15:41:42.136499 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:42.136437    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:42.136499 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:42.136491    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:46.136478833 +0000 UTC m=+65.621039408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:42.202482 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:42.202456    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf4lp"
Jul 24 15:41:45.364768 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:45.364738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:41:45.365268 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:45.364842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:45.365268 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:45.364887    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:45.365268 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:45.364912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:41:45.365268 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:45.364963    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls podName:147ab95a-ccc0-46b6-88f6-553b434f3268 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:01.364942125 +0000 UTC m=+80.849502704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-r9c4b" (UID: "147ab95a-ccc0-46b6-88f6-553b434f3268") : secret "cluster-monitoring-operator-tls" not found
Jul 24 15:41:45.365268 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:45.364988    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 15:41:45.365268 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:45.365032    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:01.365018827 +0000 UTC m=+80.849579403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : configmap references non-existent config key: service-ca.crt
Jul 24 15:41:45.365268 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:45.365053    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs podName:8814c4dc-52a8-4447-a645-adecc1b46d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:01.365046802 +0000 UTC m=+80.849607378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs") pod "router-default-798f8fd9f9-zr8tr" (UID: "8814c4dc-52a8-4447-a645-adecc1b46d89") : secret "router-metrics-certs-default" not found
Jul 24 15:41:45.466052 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:45.466016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:41:45.466270 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:45.466160    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 15:41:45.466270 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:45.466191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:41:45.466270 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:45.466212    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert podName:e25ae61d-8fd4-499c-9d04-a58302f9886a nodeName:}" failed. No retries permitted until 2026-07-24 15:42:01.466198053 +0000 UTC m=+80.950758629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-6vwr4" (UID: "e25ae61d-8fd4-499c-9d04-a58302f9886a") : secret "networking-console-plugin-cert" not found
Jul 24 15:41:45.466437 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:45.466304    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 15:41:45.466437 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:45.466373    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls podName:48b1c8e8-08ed-4097-8c80-ef6cd01541ee nodeName:}" failed. No retries permitted until 2026-07-24 15:42:01.466358165 +0000 UTC m=+80.950918744 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls") pod "cluster-samples-operator-6748489656-msxpl" (UID: "48b1c8e8-08ed-4097-8c80-ef6cd01541ee") : secret "samples-operator-tls" not found
Jul 24 15:41:46.173054 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.173015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:46.173231 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.173131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:46.173231 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:46.173165    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:46.173231 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:46.173220    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 15:41:46.173231 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:46.173228    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:54.173212387 +0000 UTC m=+73.657772964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : secret "insights-runtime-extractor-tls" not found
Jul 24 15:41:46.173408 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:46.173327    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm podName:2d245c2e-8405-4b45-927b-cd0bcfbd5883 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:54.17331268 +0000 UTC m=+73.657873255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lx822" (UID: "2d245c2e-8405-4b45-927b-cd0bcfbd5883") : configmap "kube-rbac-proxy" not found
Jul 24 15:41:46.779572 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.779538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:46.779953 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.779658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:46.782093 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.782076    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 15:41:46.782346 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.782324    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpd66\" (UniqueName: \"kubernetes.io/projected/b03c5f74-db11-4405-93ae-cd1b77d40f4a-kube-api-access-wpd66\") pod \"network-check-target-kr79g\" (UID: \"b03c5f74-db11-4405-93ae-cd1b77d40f4a\") " pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:46.792049 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.792027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33d7b0af-08cf-4fa6-9f39-e461402d71f1-original-pull-secret\") pod \"global-pull-secret-syncer-9k7ms\" (UID: \"33d7b0af-08cf-4fa6-9f39-e461402d71f1\") " pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:46.880714 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.880683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:41:46.882966 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.882950    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 15:41:46.891476 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:46.891458    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 15:41:46.891529 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:46.891518    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs podName:d71a09df-224a-475a-9c69-4fe776c7cdb3 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:50.89149711 +0000 UTC m=+130.376057685 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs") pod "network-metrics-daemon-2wl2t" (UID: "d71a09df-224a-475a-9c69-4fe776c7cdb3") : secret "metrics-daemon-secret" not found
Jul 24 15:41:46.955662 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.955628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9k7ms"
Jul 24 15:41:46.962659 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.962641    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hvmsc\""
Jul 24 15:41:46.970917 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:46.970894    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:47.090198 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.090170    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9k7ms"]
Jul 24 15:41:47.094473 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:47.094429    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d7b0af_08cf_4fa6_9f39_e461402d71f1.slice/crio-38964395a123b27fb582a178aeb63842f6dfbb05af4c0faed3d83620221bdab4 WatchSource:0}: Error finding container 38964395a123b27fb582a178aeb63842f6dfbb05af4c0faed3d83620221bdab4: Status 404 returned error can't find the container with id 38964395a123b27fb582a178aeb63842f6dfbb05af4c0faed3d83620221bdab4
Jul 24 15:41:47.106839 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.106817    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kr79g"]
Jul 24 15:41:47.109732 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:47.109704    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03c5f74_db11_4405_93ae_cd1b77d40f4a.slice/crio-deb6e06264b828503a2c049b0a25456824c9d914613a44a4cdb101cdc8655f78 WatchSource:0}: Error finding container deb6e06264b828503a2c049b0a25456824c9d914613a44a4cdb101cdc8655f78: Status 404 returned error can't find the container with id deb6e06264b828503a2c049b0a25456824c9d914613a44a4cdb101cdc8655f78
Jul 24 15:41:47.322815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.322726    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9k7ms" event={"ID":"33d7b0af-08cf-4fa6-9f39-e461402d71f1","Type":"ContainerStarted","Data":"38964395a123b27fb582a178aeb63842f6dfbb05af4c0faed3d83620221bdab4"}
Jul 24 15:41:47.324227 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.324200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kr79g" event={"ID":"b03c5f74-db11-4405-93ae-cd1b77d40f4a","Type":"ContainerStarted","Data":"19617ff347010b87aff890fba7d3789e772c3fe846abd3deb5f38cd357e1fa4a"}
Jul 24 15:41:47.324369 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.324257    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kr79g" event={"ID":"b03c5f74-db11-4405-93ae-cd1b77d40f4a","Type":"ContainerStarted","Data":"deb6e06264b828503a2c049b0a25456824c9d914613a44a4cdb101cdc8655f78"}
Jul 24 15:41:47.339592 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.339556    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kr79g" podStartSLOduration=66.339542854 podStartE2EDuration="1m6.339542854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:47.339160571 +0000 UTC m=+66.823721171" watchObservedRunningTime="2026-07-24 15:41:47.339542854 +0000 UTC m=+66.824103442"
Jul 24 15:41:47.687817 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.687780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:41:47.687988 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:47.687955    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 15:41:47.687988 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:47.687974    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d4b6fdd4-cst5j: secret "image-registry-tls" not found
Jul 24 15:41:47.688125 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:47.688033    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls podName:f7d54a2a-55ae-4f11-99db-3f98a78db125 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:19.68801139 +0000 UTC m=+99.172571970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls") pod "image-registry-79d4b6fdd4-cst5j" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125") : secret "image-registry-tls" not found
Jul 24 15:41:47.788658 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.788624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:47.791546 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.791522    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66b549b1-da81-4293-95c9-a7524f30bc40-metrics-tls\") pod \"dns-default-tf88h\" (UID: \"66b549b1-da81-4293-95c9-a7524f30bc40\") " pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:47.967305 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.967203    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dz9d4\""
Jul 24 15:41:47.975447 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:47.975418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:48.137838 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:48.137677    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tf88h"]
Jul 24 15:41:48.140552 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:48.140524    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b549b1_da81_4293_95c9_a7524f30bc40.slice/crio-0cb6e9f2c7aab4a6438507b2ce7198205df8b13f5b67483072483357794d3adb WatchSource:0}: Error finding container 0cb6e9f2c7aab4a6438507b2ce7198205df8b13f5b67483072483357794d3adb: Status 404 returned error can't find the container with id 0cb6e9f2c7aab4a6438507b2ce7198205df8b13f5b67483072483357794d3adb
Jul 24 15:41:48.328674 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:48.328590    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tf88h" event={"ID":"66b549b1-da81-4293-95c9-a7524f30bc40","Type":"ContainerStarted","Data":"0cb6e9f2c7aab4a6438507b2ce7198205df8b13f5b67483072483357794d3adb"}
Jul 24 15:41:48.328819 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:48.328738    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:41:51.999469 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:51.999437    2576 scope.go:117] "RemoveContainer" containerID="345a5b820ecb33d5383ed27e702cccc8d5d2a960bdcc7e1fbbf2c336a04febd9"
Jul 24 15:41:52.344364 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.344278    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tf88h" event={"ID":"66b549b1-da81-4293-95c9-a7524f30bc40","Type":"ContainerStarted","Data":"c24ff32032fea3dadf215dbe1e40a735e7d3187057b5868f31f57f91b659c624"}
Jul 24 15:41:52.344364 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.344326    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tf88h" event={"ID":"66b549b1-da81-4293-95c9-a7524f30bc40","Type":"ContainerStarted","Data":"709c8424352b4b545e8945c04a919b7f057f554eee4063fcd62b0646937f301f"}
Jul 24 15:41:52.344563 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.344393    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tf88h"
Jul 24 15:41:52.345744 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.345722    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9k7ms" event={"ID":"33d7b0af-08cf-4fa6-9f39-e461402d71f1","Type":"ContainerStarted","Data":"a9f9cbc4ea5b3a8fc8af1fdd11b5d553725436c77517d6162db41c7e5879c3d4"}
Jul 24 15:41:52.347068 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.347052    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 15:41:52.347423 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.347405    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/1.log"
Jul 24 15:41:52.347481 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.347442    2576 generic.go:356] "Generic (PLEG): container finished" podID="b9580587-864e-44e1-8d91-33cc9aa83cc5" containerID="e9e201ca861e742608e6061b632b4681a10c97ca5c40c330a6da0905a1546bc2" exitCode=255
Jul 24 15:41:52.347481 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.347471    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" event={"ID":"b9580587-864e-44e1-8d91-33cc9aa83cc5","Type":"ContainerDied","Data":"e9e201ca861e742608e6061b632b4681a10c97ca5c40c330a6da0905a1546bc2"}
Jul 24 15:41:52.347541 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.347504    2576 scope.go:117] "RemoveContainer" containerID="345a5b820ecb33d5383ed27e702cccc8d5d2a960bdcc7e1fbbf2c336a04febd9"
Jul 24 15:41:52.347761 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.347744    2576 scope.go:117] "RemoveContainer" containerID="e9e201ca861e742608e6061b632b4681a10c97ca5c40c330a6da0905a1546bc2"
Jul 24 15:41:52.347906 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:52.347889    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-9b8th_openshift-console-operator(b9580587-864e-44e1-8d91-33cc9aa83cc5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" podUID="b9580587-864e-44e1-8d91-33cc9aa83cc5"
Jul 24 15:41:52.361638 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.361605    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tf88h" podStartSLOduration=34.141680866 podStartE2EDuration="37.361594296s" totalImagesPullingTime="3.21991343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:52.360352356 +0000 UTC m=+71.844912955" watchObservedRunningTime="2026-07-24 15:41:52.361594296 +0000 UTC m=+71.846154894"
Jul 24 15:41:52.373881 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:52.373842    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9k7ms" podStartSLOduration=67.097370219 podStartE2EDuration="1m11.373830614s" totalImagesPullingTime="4.276460395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:52.37334454 +0000 UTC m=+71.857905147" watchObservedRunningTime="2026-07-24 15:41:52.373830614 +0000 UTC m=+71.858391212"
Jul 24 15:41:53.354429 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:53.354404    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 15:41:54.248699 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:54.248661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:54.248916 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:54.248756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:54.249279 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:54.249215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d245c2e-8405-4b45-927b-cd0bcfbd5883-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:54.251228 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:54.251211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d245c2e-8405-4b45-927b-cd0bcfbd5883-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx822\" (UID: \"2d245c2e-8405-4b45-927b-cd0bcfbd5883\") " pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:54.348890 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:54.348858    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtzxv\""
Jul 24 15:41:54.355695 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:54.355661    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lx822"
Jul 24 15:41:54.486320 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:54.486275    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lx822"]
Jul 24 15:41:54.489155 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:54.489125    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d245c2e_8405_4b45_927b_cd0bcfbd5883.slice/crio-a0ae247d22f01c92cd94d5a4a92308bd0888bf68444cc44d95b46f054540d829 WatchSource:0}: Error finding container a0ae247d22f01c92cd94d5a4a92308bd0888bf68444cc44d95b46f054540d829: Status 404 returned error can't find the container with id a0ae247d22f01c92cd94d5a4a92308bd0888bf68444cc44d95b46f054540d829
Jul 24 15:41:55.364003 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:55.363963    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx822" event={"ID":"2d245c2e-8405-4b45-927b-cd0bcfbd5883","Type":"ContainerStarted","Data":"a0ae247d22f01c92cd94d5a4a92308bd0888bf68444cc44d95b46f054540d829"}
Jul 24 15:41:57.981485 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:57.981443    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"]
Jul 24 15:41:58.002132 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.002098    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"]
Jul 24 15:41:58.002334 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.002212    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.006342 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.006314    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 24 15:41:58.006529 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.006386    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 15:41:58.006529 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.006463    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 15:41:58.006529 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.006391    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 15:41:58.186672 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.186635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/41f41d7c-a03c-4316-a336-f846a82f9b56-klusterlet-config\") pod \"klusterlet-addon-workmgr-75d848ff75-t7kq5\" (UID: \"41f41d7c-a03c-4316-a336-f846a82f9b56\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.186859 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.186761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdcxq\" (UniqueName: \"kubernetes.io/projected/41f41d7c-a03c-4316-a336-f846a82f9b56-kube-api-access-hdcxq\") pod \"klusterlet-addon-workmgr-75d848ff75-t7kq5\" (UID: \"41f41d7c-a03c-4316-a336-f846a82f9b56\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.186859 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.186855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41f41d7c-a03c-4316-a336-f846a82f9b56-tmp\") pod \"klusterlet-addon-workmgr-75d848ff75-t7kq5\" (UID: \"41f41d7c-a03c-4316-a336-f846a82f9b56\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.287624 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.287530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdcxq\" (UniqueName: \"kubernetes.io/projected/41f41d7c-a03c-4316-a336-f846a82f9b56-kube-api-access-hdcxq\") pod \"klusterlet-addon-workmgr-75d848ff75-t7kq5\" (UID: \"41f41d7c-a03c-4316-a336-f846a82f9b56\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.287808 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.287647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41f41d7c-a03c-4316-a336-f846a82f9b56-tmp\") pod \"klusterlet-addon-workmgr-75d848ff75-t7kq5\" (UID: \"41f41d7c-a03c-4316-a336-f846a82f9b56\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.287808 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.287714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/41f41d7c-a03c-4316-a336-f846a82f9b56-klusterlet-config\") pod \"klusterlet-addon-workmgr-75d848ff75-t7kq5\" (UID: \"41f41d7c-a03c-4316-a336-f846a82f9b56\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.288080 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.288057    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41f41d7c-a03c-4316-a336-f846a82f9b56-tmp\") pod \"klusterlet-addon-workmgr-75d848ff75-t7kq5\" (UID: \"41f41d7c-a03c-4316-a336-f846a82f9b56\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.290969 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.290936    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/41f41d7c-a03c-4316-a336-f846a82f9b56-klusterlet-config\") pod \"klusterlet-addon-workmgr-75d848ff75-t7kq5\" (UID: \"41f41d7c-a03c-4316-a336-f846a82f9b56\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.299481 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.299453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdcxq\" (UniqueName: \"kubernetes.io/projected/41f41d7c-a03c-4316-a336-f846a82f9b56-kube-api-access-hdcxq\") pod \"klusterlet-addon-workmgr-75d848ff75-t7kq5\" (UID: \"41f41d7c-a03c-4316-a336-f846a82f9b56\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:58.314213 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:58.314131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:41:59.417635 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:59.417612    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"]
Jul 24 15:41:59.419933 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:41:59.419903    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f41d7c_a03c_4316_a336_f846a82f9b56.slice/crio-948daea55179b58ee22972378254b26188836c708ff5cd484e6a23ea14f721ba WatchSource:0}: Error finding container 948daea55179b58ee22972378254b26188836c708ff5cd484e6a23ea14f721ba: Status 404 returned error can't find the container with id 948daea55179b58ee22972378254b26188836c708ff5cd484e6a23ea14f721ba
Jul 24 15:41:59.758788 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:59.758754    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:59.758946 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:59.758812    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:41:59.759140 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:41:59.759126    2576 scope.go:117] "RemoveContainer" containerID="e9e201ca861e742608e6061b632b4681a10c97ca5c40c330a6da0905a1546bc2"
Jul 24 15:41:59.759306 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:41:59.759289    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-9b8th_openshift-console-operator(b9580587-864e-44e1-8d91-33cc9aa83cc5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" podUID="b9580587-864e-44e1-8d91-33cc9aa83cc5"
Jul 24 15:42:00.386909 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:00.386669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5" event={"ID":"41f41d7c-a03c-4316-a336-f846a82f9b56","Type":"ContainerStarted","Data":"948daea55179b58ee22972378254b26188836c708ff5cd484e6a23ea14f721ba"}
Jul 24 15:42:00.388995 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:00.388891    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx822" event={"ID":"2d245c2e-8405-4b45-927b-cd0bcfbd5883","Type":"ContainerStarted","Data":"e91ff8bbe707a1c35a7f22eb62248af8cac25a68d1338336e7a9979bcf08397c"}
Jul 24 15:42:01.424308 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.424272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:42:01.424759 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.424382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:42:01.424759 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.424456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:42:01.425036 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.424985    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8814c4dc-52a8-4447-a645-adecc1b46d89-service-ca-bundle\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:42:01.427737 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.427710    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/147ab95a-ccc0-46b6-88f6-553b434f3268-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-r9c4b\" (UID: \"147ab95a-ccc0-46b6-88f6-553b434f3268\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:42:01.427850 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.427733    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8814c4dc-52a8-4447-a645-adecc1b46d89-metrics-certs\") pod \"router-default-798f8fd9f9-zr8tr\" (UID: \"8814c4dc-52a8-4447-a645-adecc1b46d89\") " pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:42:01.525070 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.525032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:42:01.525235 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.525173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:42:01.527870 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.527843    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e25ae61d-8fd4-499c-9d04-a58302f9886a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6vwr4\" (UID: \"e25ae61d-8fd4-499c-9d04-a58302f9886a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:42:01.527994 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.527878    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48b1c8e8-08ed-4097-8c80-ef6cd01541ee-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-msxpl\" (UID: \"48b1c8e8-08ed-4097-8c80-ef6cd01541ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:42:01.612843 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.612813    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jm9fp\""
Jul 24 15:42:01.621005 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.620970    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:42:01.665101 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.664905    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pgts6\""
Jul 24 15:42:01.672919 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.672894    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"
Jul 24 15:42:01.738076 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.738043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2bj9d\""
Jul 24 15:42:01.745555 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.745526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"
Jul 24 15:42:01.757474 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.757450    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9f6dk\""
Jul 24 15:42:01.764672 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:01.764646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"
Jul 24 15:42:02.357376 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:02.357345    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tf88h"
Jul 24 15:42:05.114557 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.114527    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl"]
Jul 24 15:42:05.349528 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.349493    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4"]
Jul 24 15:42:05.350619 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.350596    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b"]
Jul 24 15:42:05.352801 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.352769    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-798f8fd9f9-zr8tr"]
Jul 24 15:42:05.353099 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:42:05.353072    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25ae61d_8fd4_499c_9d04_a58302f9886a.slice/crio-69755781a4b8cb6b2cd4a636bac2a180220d601c58c988d2e4ec3bcf0cf5f760 WatchSource:0}: Error finding container 69755781a4b8cb6b2cd4a636bac2a180220d601c58c988d2e4ec3bcf0cf5f760: Status 404 returned error can't find the container with id 69755781a4b8cb6b2cd4a636bac2a180220d601c58c988d2e4ec3bcf0cf5f760
Jul 24 15:42:05.355581 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:42:05.354805    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147ab95a_ccc0_46b6_88f6_553b434f3268.slice/crio-c4b25f1680d59ecb1bb915f04cd716d1242db3f674af838903d66622520062f9 WatchSource:0}: Error finding container c4b25f1680d59ecb1bb915f04cd716d1242db3f674af838903d66622520062f9: Status 404 returned error can't find the container with id c4b25f1680d59ecb1bb915f04cd716d1242db3f674af838903d66622520062f9
Jul 24 15:42:05.357965 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:42:05.357731    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8814c4dc_52a8_4447_a645_adecc1b46d89.slice/crio-92a52e59c1eb74c1b0255f8ae30ba6d1e9e8fac833cbee7fa0e439504bfd6fcc WatchSource:0}: Error finding container 92a52e59c1eb74c1b0255f8ae30ba6d1e9e8fac833cbee7fa0e439504bfd6fcc: Status 404 returned error can't find the container with id 92a52e59c1eb74c1b0255f8ae30ba6d1e9e8fac833cbee7fa0e439504bfd6fcc
Jul 24 15:42:05.409690 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.409663    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl" event={"ID":"48b1c8e8-08ed-4097-8c80-ef6cd01541ee","Type":"ContainerStarted","Data":"d2b613f5e3cd62396a14fe97e31347edb066092fc953022c9b8d3ffaf0a5cd4b"}
Jul 24 15:42:05.411084 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.411055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4" event={"ID":"e25ae61d-8fd4-499c-9d04-a58302f9886a","Type":"ContainerStarted","Data":"69755781a4b8cb6b2cd4a636bac2a180220d601c58c988d2e4ec3bcf0cf5f760"}
Jul 24 15:42:05.412952 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.412930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx822" event={"ID":"2d245c2e-8405-4b45-927b-cd0bcfbd5883","Type":"ContainerStarted","Data":"80994dd16020edf3a663da7a3fed4f60cb3b508de2ad545ede61207b08e65823"}
Jul 24 15:42:05.414779 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.414760    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-798f8fd9f9-zr8tr" event={"ID":"8814c4dc-52a8-4447-a645-adecc1b46d89","Type":"ContainerStarted","Data":"92a52e59c1eb74c1b0255f8ae30ba6d1e9e8fac833cbee7fa0e439504bfd6fcc"}
Jul 24 15:42:05.416491 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.416465    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5" event={"ID":"41f41d7c-a03c-4316-a336-f846a82f9b56","Type":"ContainerStarted","Data":"888d70a4918e87c5fd6e0b623171fadb38bd6d99fe47264cc036bd42632b433f"}
Jul 24 15:42:05.416699 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.416683    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:42:05.417805 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.417758    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b" event={"ID":"147ab95a-ccc0-46b6-88f6-553b434f3268","Type":"ContainerStarted","Data":"c4b25f1680d59ecb1bb915f04cd716d1242db3f674af838903d66622520062f9"}
Jul 24 15:42:05.421025 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.421004    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5"
Jul 24 15:42:05.434518 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:05.434466    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d848ff75-t7kq5" podStartSLOduration=2.912019946 podStartE2EDuration="8.434447199s" totalImagesPullingTime="5.522427253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:05.433504588 +0000 UTC m=+84.918065187" watchObservedRunningTime="2026-07-24 15:42:05.434447199 +0000 UTC m=+84.919007797"
Jul 24 15:42:06.425570 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:06.425527    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-798f8fd9f9-zr8tr" event={"ID":"8814c4dc-52a8-4447-a645-adecc1b46d89","Type":"ContainerStarted","Data":"0292cb79f03ca3478bd4eee10f3c18e3404724ca586b4e7ef11db6fec1b371f2"}
Jul 24 15:42:06.622139 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:06.622056    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:42:06.624855 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:06.624831    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:42:06.643324 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:06.643256    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-798f8fd9f9-zr8tr" podStartSLOduration=37.643216651 podStartE2EDuration="37.643216651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:06.44569845 +0000 UTC m=+85.930259049" watchObservedRunningTime="2026-07-24 15:42:06.643216651 +0000 UTC m=+86.127777250"
Jul 24 15:42:07.430064 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:07.430036    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:42:07.433042 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:07.432846    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-798f8fd9f9-zr8tr"
Jul 24 15:42:11.446050 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:11.446014    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl" event={"ID":"48b1c8e8-08ed-4097-8c80-ef6cd01541ee","Type":"ContainerStarted","Data":"8a687040757019ae9b4c1548e8aeb0eecd86d7b0f907d94e58778f531e2c3dd1"}
Jul 24 15:42:11.446566 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:11.446074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl" event={"ID":"48b1c8e8-08ed-4097-8c80-ef6cd01541ee","Type":"ContainerStarted","Data":"6cb4216c0c9a4b133c1740f9d649c05bad2c56d6dcf755c56fd7dac0fd1ba876"}
Jul 24 15:42:11.447764 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:11.447736    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4" event={"ID":"e25ae61d-8fd4-499c-9d04-a58302f9886a","Type":"ContainerStarted","Data":"8e8c1c7b920a5879646c394b7ab3bda34fa307180c1c38ffe2611de36ea463f0"}
Jul 24 15:42:11.449767 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:11.449728    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx822" event={"ID":"2d245c2e-8405-4b45-927b-cd0bcfbd5883","Type":"ContainerStarted","Data":"f18391343b825fdd91ee75effd78fef0141798a77a1439b0fad5c9c4805988d5"}
Jul 24 15:42:11.451091 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:11.451069    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b" event={"ID":"147ab95a-ccc0-46b6-88f6-553b434f3268","Type":"ContainerStarted","Data":"55cdb626464ad42a6f034117939c64ffbb5af54570cc1a3911c9af3d2e9fd6a6"}
Jul 24 15:42:11.463355 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:11.463300    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-msxpl" podStartSLOduration=37.068432104 podStartE2EDuration="42.463282133s" totalImagesPullingTime="5.394850029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:11.461808032 +0000 UTC m=+90.946368637" watchObservedRunningTime="2026-07-24 15:42:11.463282133 +0000 UTC m=+90.947842729"
Jul 24 15:42:11.477414 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:11.477361    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6vwr4" podStartSLOduration=37.272318748 podStartE2EDuration="42.477324949s" totalImagesPullingTime="5.205006201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:11.476870463 +0000 UTC m=+90.961431063" watchObservedRunningTime="2026-07-24 15:42:11.477324949 +0000 UTC m=+90.961885547"
Jul 24 15:42:11.494260 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:11.494180    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-r9c4b" podStartSLOduration=37.287517402 podStartE2EDuration="42.494160562s" totalImagesPullingTime="5.20664316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:11.493100331 +0000 UTC m=+90.977660929" watchObservedRunningTime="2026-07-24 15:42:11.494160562 +0000 UTC m=+90.978721161"
Jul 24 15:42:11.514536 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:11.514469    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lx822" podStartSLOduration=17.622634367 podStartE2EDuration="33.514449146s" totalImagesPullingTime="15.891814779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:11.511045223 +0000 UTC m=+90.995605822" watchObservedRunningTime="2026-07-24 15:42:11.514449146 +0000 UTC m=+90.999009744"
Jul 24 15:42:13.004417 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:13.004391    2576 scope.go:117] "RemoveContainer" containerID="e9e201ca861e742608e6061b632b4681a10c97ca5c40c330a6da0905a1546bc2"
Jul 24 15:42:13.460992 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:13.460966    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 15:42:13.461173 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:13.461016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" event={"ID":"b9580587-864e-44e1-8d91-33cc9aa83cc5","Type":"ContainerStarted","Data":"7a44b80b65cef6da003b1e035372cd1bed67062a59786461f7b93b5e05265ddf"}
Jul 24 15:42:13.461330 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:13.461310    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:42:13.477690 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:13.477640    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th" podStartSLOduration=38.057108062 podStartE2EDuration="44.477625011s" totalImagesPullingTime="6.420516949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:13.477126586 +0000 UTC m=+92.961687205" watchObservedRunningTime="2026-07-24 15:42:13.477625011 +0000 UTC m=+92.962185608"
Jul 24 15:42:13.991294 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:13.991263    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-9b8th"
Jul 24 15:42:19.335988 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:19.335959    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kr79g"
Jul 24 15:42:19.778930 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:19.778897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:42:19.781380 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:19.781355    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"image-registry-79d4b6fdd4-cst5j\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") " pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:42:20.036842 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:20.036756    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wlmxz\""
Jul 24 15:42:20.044749 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:20.044724    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:42:20.176592 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:20.176405    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79d4b6fdd4-cst5j"]
Jul 24 15:42:20.179122 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:42:20.179096    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d54a2a_55ae_4f11_99db_3f98a78db125.slice/crio-fe46f229f0cf15badefb3792ba772c18e0c7259ca6bfb8bf2ce5dfcf191b2fe9 WatchSource:0}: Error finding container fe46f229f0cf15badefb3792ba772c18e0c7259ca6bfb8bf2ce5dfcf191b2fe9: Status 404 returned error can't find the container with id fe46f229f0cf15badefb3792ba772c18e0c7259ca6bfb8bf2ce5dfcf191b2fe9
Jul 24 15:42:20.488589 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:20.488557    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j" event={"ID":"f7d54a2a-55ae-4f11-99db-3f98a78db125","Type":"ContainerStarted","Data":"ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951"}
Jul 24 15:42:20.489024 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:20.488609    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j" event={"ID":"f7d54a2a-55ae-4f11-99db-3f98a78db125","Type":"ContainerStarted","Data":"fe46f229f0cf15badefb3792ba772c18e0c7259ca6bfb8bf2ce5dfcf191b2fe9"}
Jul 24 15:42:20.489024 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:20.488638    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:42:20.509946 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:20.509889    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j" podStartSLOduration=99.509873704 podStartE2EDuration="1m39.509873704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:20.508024014 +0000 UTC m=+99.992584634" watchObservedRunningTime="2026-07-24 15:42:20.509873704 +0000 UTC m=+99.994434303"
Jul 24 15:42:21.455749 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:21.455713    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79d4b6fdd4-cst5j"]
Jul 24 15:42:23.638025 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.637989    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zpkwd"]
Jul 24 15:42:23.661736 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.661710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.664344 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.664321    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bk4xx\""
Jul 24 15:42:23.664546 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.664533    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 15:42:23.664805 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.664791    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 15:42:23.664846 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.664799    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 15:42:23.665581 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.665562    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 15:42:23.711359 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.711330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-wtmp\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.711543 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.711372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92032438-e0c7-4d52-836d-236573ff3419-sys\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.711543 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.711392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92032438-e0c7-4d52-836d-236573ff3419-root\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.711543 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.711411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx74l\" (UniqueName: \"kubernetes.io/projected/92032438-e0c7-4d52-836d-236573ff3419-kube-api-access-fx74l\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.711543 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.711491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.711682 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.711568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-tls\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.711682 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.711590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92032438-e0c7-4d52-836d-236573ff3419-metrics-client-ca\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.711682 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.711609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-textfile\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.711682 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.711633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812208 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-textfile\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812208 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-wtmp\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92032438-e0c7-4d52-836d-236573ff3419-sys\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92032438-e0c7-4d52-836d-236573ff3419-root\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx74l\" (UniqueName: \"kubernetes.io/projected/92032438-e0c7-4d52-836d-236573ff3419-kube-api-access-fx74l\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812427    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92032438-e0c7-4d52-836d-236573ff3419-sys\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812473 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-wtmp\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812820 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-tls\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812820 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812547    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92032438-e0c7-4d52-836d-236573ff3419-metrics-client-ca\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812820 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:42:23.812610    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 15:42:23.812820 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812633    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-textfile\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.812820 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:42:23.812672    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-tls podName:92032438-e0c7-4d52-836d-236573ff3419 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:24.312656991 +0000 UTC m=+103.797217567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-tls") pod "node-exporter-zpkwd" (UID: "92032438-e0c7-4d52-836d-236573ff3419") : secret "node-exporter-tls" not found
Jul 24 15:42:23.812820 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.812696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92032438-e0c7-4d52-836d-236573ff3419-root\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.813260 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.813222    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.813466 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.813446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92032438-e0c7-4d52-836d-236573ff3419-metrics-client-ca\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.815957 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.815933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:23.824539 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:23.824506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx74l\" (UniqueName: \"kubernetes.io/projected/92032438-e0c7-4d52-836d-236573ff3419-kube-api-access-fx74l\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:24.316582 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:24.316545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-tls\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:24.319058 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:24.319032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92032438-e0c7-4d52-836d-236573ff3419-node-exporter-tls\") pod \"node-exporter-zpkwd\" (UID: \"92032438-e0c7-4d52-836d-236573ff3419\") " pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:24.571907 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:24.571832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zpkwd"
Jul 24 15:42:24.582426 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:42:24.582397    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92032438_e0c7_4d52_836d_236573ff3419.slice/crio-a01018791ac33bcc0c1dd1410f453a55d8039827a88b4745244f20c1f8a4672e WatchSource:0}: Error finding container a01018791ac33bcc0c1dd1410f453a55d8039827a88b4745244f20c1f8a4672e: Status 404 returned error can't find the container with id a01018791ac33bcc0c1dd1410f453a55d8039827a88b4745244f20c1f8a4672e
Jul 24 15:42:25.508080 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:25.508046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpkwd" event={"ID":"92032438-e0c7-4d52-836d-236573ff3419","Type":"ContainerStarted","Data":"a01018791ac33bcc0c1dd1410f453a55d8039827a88b4745244f20c1f8a4672e"}
Jul 24 15:42:26.514086 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:26.514055    2576 generic.go:356] "Generic (PLEG): container finished" podID="92032438-e0c7-4d52-836d-236573ff3419" containerID="6c08fdecc627ac3c40bf1fbc5d807379adb6d750b558c2c235aceccf56168f20" exitCode=0
Jul 24 15:42:26.514558 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:26.514126    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpkwd" event={"ID":"92032438-e0c7-4d52-836d-236573ff3419","Type":"ContainerDied","Data":"6c08fdecc627ac3c40bf1fbc5d807379adb6d750b558c2c235aceccf56168f20"}
Jul 24 15:42:27.520309 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:27.520274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpkwd" event={"ID":"92032438-e0c7-4d52-836d-236573ff3419","Type":"ContainerStarted","Data":"4166883f003cf9a83801fa8fceab0eb3cd62242162416239ff7518fdfc41ee4a"}
Jul 24 15:42:27.520688 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:27.520333    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpkwd" event={"ID":"92032438-e0c7-4d52-836d-236573ff3419","Type":"ContainerStarted","Data":"104f861b91a787c072bbcd8d736a23aeb95b6b0bcb625d99c887191a2d229418"}
Jul 24 15:42:27.544612 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:27.544558    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zpkwd" podStartSLOduration=3.508625222 podStartE2EDuration="4.544542783s" totalImagesPullingTime="963.183012ms" totalInitContainerRuntime="72.734549ms" isStatefulPod=true podCreationTimestamp="2026-07-24 15:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:27.543235979 +0000 UTC m=+107.027796576" watchObservedRunningTime="2026-07-24 15:42:27.544542783 +0000 UTC m=+107.029103358"
Jul 24 15:42:32.874630 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.874593    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66965b8bcb-wllnh"]
Jul 24 15:42:32.908004 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.907966    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66965b8bcb-wllnh"]
Jul 24 15:42:32.908164 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.908072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:32.911072 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.911047    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vx7mq\""
Jul 24 15:42:32.911350 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.911334    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 15:42:32.911412 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.911394    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 15:42:32.912308 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.912293    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 15:42:32.912401 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.912355    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 15:42:32.912479 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.912461    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 15:42:32.912542 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.912508    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 15:42:32.912542 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.912514    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 15:42:32.916699 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.916681    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 15:42:32.991517 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.991487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wf6\" (UniqueName: \"kubernetes.io/projected/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-kube-api-access-66wf6\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:32.991653 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.991524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-trusted-ca-bundle\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:32.991653 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.991596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-config\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:32.991653 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.991640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-service-ca\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:32.991754 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.991675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-serving-cert\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:32.991754 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.991720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-oauth-config\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:32.991754 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:32.991737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-oauth-serving-cert\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.092912 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.092877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-oauth-serving-cert\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093062 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.092922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wf6\" (UniqueName: \"kubernetes.io/projected/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-kube-api-access-66wf6\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093062 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.092943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-trusted-ca-bundle\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093139 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.093116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-config\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093192 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.093177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-service-ca\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093262 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.093214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-serving-cert\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093351 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.093335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-oauth-config\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.093600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-oauth-serving-cert\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093706 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.093695    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-trusted-ca-bundle\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093864 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.093842    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-config\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.093902 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.093871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-service-ca\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.095843 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.095822    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-serving-cert\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.095923 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.095867    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-oauth-config\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.101184 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.101156    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wf6\" (UniqueName: \"kubernetes.io/projected/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-kube-api-access-66wf6\") pod \"console-66965b8bcb-wllnh\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") " pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.218174 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.218138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:33.394086 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.394056    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66965b8bcb-wllnh"]
Jul 24 15:42:33.397080 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:42:33.397053    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd186fad1_0cf3_43b1_9d7d_e51cf5ea0458.slice/crio-152a567b9aefd7b8dad6c04029ae8f46d4d206692f593ea2eb2309fde4597e53 WatchSource:0}: Error finding container 152a567b9aefd7b8dad6c04029ae8f46d4d206692f593ea2eb2309fde4597e53: Status 404 returned error can't find the container with id 152a567b9aefd7b8dad6c04029ae8f46d4d206692f593ea2eb2309fde4597e53
Jul 24 15:42:33.544686 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:33.544611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66965b8bcb-wllnh" event={"ID":"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458","Type":"ContainerStarted","Data":"152a567b9aefd7b8dad6c04029ae8f46d4d206692f593ea2eb2309fde4597e53"}
Jul 24 15:42:37.569964 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:37.569920    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66965b8bcb-wllnh" event={"ID":"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458","Type":"ContainerStarted","Data":"808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817"}
Jul 24 15:42:37.588041 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:37.587988    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66965b8bcb-wllnh" podStartSLOduration=2.416525618 podStartE2EDuration="5.587968876s" totalImagesPullingTime="3.171443258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:37.586640852 +0000 UTC m=+117.071201431" watchObservedRunningTime="2026-07-24 15:42:37.587968876 +0000 UTC m=+117.072529474"
Jul 24 15:42:42.502594 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:42.502564    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:42:43.219249 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:43.219216    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:43.219413 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:43.219298    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:43.224167 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:43.224142    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:43.595996 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:43.595918    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:42:47.516497 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.516449    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j" podUID="f7d54a2a-55ae-4f11-99db-3f98a78db125" containerName="registry" containerID="cri-o://ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951" gracePeriod=30
Jul 24 15:42:47.755724 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.755697    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:42:47.921801 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.921761    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-trusted-ca\") pod \"f7d54a2a-55ae-4f11-99db-3f98a78db125\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") "
Jul 24 15:42:47.922002 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.921812    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-certificates\") pod \"f7d54a2a-55ae-4f11-99db-3f98a78db125\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") "
Jul 24 15:42:47.922002 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.921839    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-installation-pull-secrets\") pod \"f7d54a2a-55ae-4f11-99db-3f98a78db125\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") "
Jul 24 15:42:47.922002 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.921884    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d54a2a-55ae-4f11-99db-3f98a78db125-ca-trust-extracted\") pod \"f7d54a2a-55ae-4f11-99db-3f98a78db125\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") "
Jul 24 15:42:47.922002 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.921914    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") pod \"f7d54a2a-55ae-4f11-99db-3f98a78db125\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") "
Jul 24 15:42:47.922002 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.921976    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-bound-sa-token\") pod \"f7d54a2a-55ae-4f11-99db-3f98a78db125\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") "
Jul 24 15:42:47.922278 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.922021    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wxl7\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-kube-api-access-4wxl7\") pod \"f7d54a2a-55ae-4f11-99db-3f98a78db125\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") "
Jul 24 15:42:47.922278 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.922071    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-image-registry-private-configuration\") pod \"f7d54a2a-55ae-4f11-99db-3f98a78db125\" (UID: \"f7d54a2a-55ae-4f11-99db-3f98a78db125\") "
Jul 24 15:42:47.922376 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.922295    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f7d54a2a-55ae-4f11-99db-3f98a78db125" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:42:47.922376 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.922309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f7d54a2a-55ae-4f11-99db-3f98a78db125" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:42:47.924667 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.924616    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-kube-api-access-4wxl7" (OuterVolumeSpecName: "kube-api-access-4wxl7") pod "f7d54a2a-55ae-4f11-99db-3f98a78db125" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125"). InnerVolumeSpecName "kube-api-access-4wxl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:42:47.924815 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.924764    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f7d54a2a-55ae-4f11-99db-3f98a78db125" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:42:47.924913 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.924894    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f7d54a2a-55ae-4f11-99db-3f98a78db125" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:42:47.924989 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.924969    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f7d54a2a-55ae-4f11-99db-3f98a78db125" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:42:47.925057 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.925038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f7d54a2a-55ae-4f11-99db-3f98a78db125" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:42:47.930946 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:47.930920    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d54a2a-55ae-4f11-99db-3f98a78db125-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f7d54a2a-55ae-4f11-99db-3f98a78db125" (UID: "f7d54a2a-55ae-4f11-99db-3f98a78db125"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:42:48.023192 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.023155    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-bound-sa-token\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.023192 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.023185    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxl7\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-kube-api-access-4wxl7\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.023192 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.023198    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-image-registry-private-configuration\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.023423 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.023207    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-trusted-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.023423 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.023216    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-certificates\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.023423 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.023224    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d54a2a-55ae-4f11-99db-3f98a78db125-installation-pull-secrets\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.023423 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.023233    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d54a2a-55ae-4f11-99db-3f98a78db125-ca-trust-extracted\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.023423 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.023268    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d54a2a-55ae-4f11-99db-3f98a78db125-registry-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:42:48.611125 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.611087    2576 generic.go:356] "Generic (PLEG): container finished" podID="f7d54a2a-55ae-4f11-99db-3f98a78db125" containerID="ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951" exitCode=0
Jul 24 15:42:48.611606 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.611151    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j"
Jul 24 15:42:48.611606 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.611177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j" event={"ID":"f7d54a2a-55ae-4f11-99db-3f98a78db125","Type":"ContainerDied","Data":"ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951"}
Jul 24 15:42:48.611606 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.611229    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d4b6fdd4-cst5j" event={"ID":"f7d54a2a-55ae-4f11-99db-3f98a78db125","Type":"ContainerDied","Data":"fe46f229f0cf15badefb3792ba772c18e0c7259ca6bfb8bf2ce5dfcf191b2fe9"}
Jul 24 15:42:48.611606 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.611275    2576 scope.go:117] "RemoveContainer" containerID="ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951"
Jul 24 15:42:48.619994 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.619977    2576 scope.go:117] "RemoveContainer" containerID="ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951"
Jul 24 15:42:48.620302 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:42:48.620226    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951\": container with ID starting with ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951 not found: ID does not exist" containerID="ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951"
Jul 24 15:42:48.620302 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.620270    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951"} err="failed to get container status \"ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951\": rpc error: code = NotFound desc = could not find container \"ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951\": container with ID starting with ffbef482de0e59176541daf31af578c7ef4fec71abe50d7420e86e4b12031951 not found: ID does not exist"
Jul 24 15:42:48.632518 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.632495    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79d4b6fdd4-cst5j"]
Jul 24 15:42:48.635912 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:48.635892    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79d4b6fdd4-cst5j"]
Jul 24 15:42:49.004110 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:49.004074    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d54a2a-55ae-4f11-99db-3f98a78db125" path="/var/lib/kubelet/pods/f7d54a2a-55ae-4f11-99db-3f98a78db125/volumes"
Jul 24 15:42:50.949543 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:50.949502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:42:50.952134 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:50.952110    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71a09df-224a-475a-9c69-4fe776c7cdb3-metrics-certs\") pod \"network-metrics-daemon-2wl2t\" (UID: \"d71a09df-224a-475a-9c69-4fe776c7cdb3\") " pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:42:51.152800 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:51.152772    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nh5k9\""
Jul 24 15:42:51.161063 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:51.161045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wl2t"
Jul 24 15:42:51.292053 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:51.292025    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2wl2t"]
Jul 24 15:42:51.296459 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:42:51.296433    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71a09df_224a_475a_9c69_4fe776c7cdb3.slice/crio-6cc9ee5f220a9d8a84cea316e9c6e21670731cd060e4b794cc61630413443b73 WatchSource:0}: Error finding container 6cc9ee5f220a9d8a84cea316e9c6e21670731cd060e4b794cc61630413443b73: Status 404 returned error can't find the container with id 6cc9ee5f220a9d8a84cea316e9c6e21670731cd060e4b794cc61630413443b73
Jul 24 15:42:51.631075 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:51.630975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wl2t" event={"ID":"d71a09df-224a-475a-9c69-4fe776c7cdb3","Type":"ContainerStarted","Data":"6cc9ee5f220a9d8a84cea316e9c6e21670731cd060e4b794cc61630413443b73"}
Jul 24 15:42:52.636676 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:52.636589    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wl2t" event={"ID":"d71a09df-224a-475a-9c69-4fe776c7cdb3","Type":"ContainerStarted","Data":"45db5c9ce6c2eeec56acd2df819d9f8481bec4405b47e16af50188e4d5714473"}
Jul 24 15:42:52.636676 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:52.636648    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wl2t" event={"ID":"d71a09df-224a-475a-9c69-4fe776c7cdb3","Type":"ContainerStarted","Data":"b786382d8dede90e008a64b69e554afa5a227148a191ee82177f02d222190b9f"}
Jul 24 15:42:52.653348 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:52.653295    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2wl2t" podStartSLOduration=130.744358627 podStartE2EDuration="2m11.653278893s" totalImagesPullingTime="908.920266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:52.652046909 +0000 UTC m=+132.136607506" watchObservedRunningTime="2026-07-24 15:42:52.653278893 +0000 UTC m=+132.137839490"
Jul 24 15:42:57.658171 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:57.658139    2576 generic.go:356] "Generic (PLEG): container finished" podID="2b3bbffa-0d24-4689-82a2-dc659dd0647a" containerID="42eabc5a9320b1a0e0085a5649cd8331db4358a585f863f0a2d2c276cee26fb6" exitCode=0
Jul 24 15:42:57.658622 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:57.658211    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk" event={"ID":"2b3bbffa-0d24-4689-82a2-dc659dd0647a","Type":"ContainerDied","Data":"42eabc5a9320b1a0e0085a5649cd8331db4358a585f863f0a2d2c276cee26fb6"}
Jul 24 15:42:57.658622 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:57.658591    2576 scope.go:117] "RemoveContainer" containerID="42eabc5a9320b1a0e0085a5649cd8331db4358a585f863f0a2d2c276cee26fb6"
Jul 24 15:42:57.659842 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:57.659732    2576 generic.go:356] "Generic (PLEG): container finished" podID="a9c1f7cf-fbd5-4107-903d-a6538f15c309" containerID="36202b48f2f10dffb9f1dc92635ffeeec62736349223b4394600ed373e6a7755" exitCode=0
Jul 24 15:42:57.659842 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:57.659803    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7" event={"ID":"a9c1f7cf-fbd5-4107-903d-a6538f15c309","Type":"ContainerDied","Data":"36202b48f2f10dffb9f1dc92635ffeeec62736349223b4394600ed373e6a7755"}
Jul 24 15:42:57.660046 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:57.660034    2576 scope.go:117] "RemoveContainer" containerID="36202b48f2f10dffb9f1dc92635ffeeec62736349223b4394600ed373e6a7755"
Jul 24 15:42:58.665539 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:58.665498    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j7ffk" event={"ID":"2b3bbffa-0d24-4689-82a2-dc659dd0647a","Type":"ContainerStarted","Data":"cd8314762ffa3bc391883a2c4bed36b05bfd593369d32b568a7c9df409d38de1"}
Jul 24 15:42:58.667235 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:42:58.667215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7" event={"ID":"a9c1f7cf-fbd5-4107-903d-a6538f15c309","Type":"ContainerStarted","Data":"3d220ff937e32db528b6e88f50e076e8c01b94ffdfe4467662af36904740861b"}
Jul 24 15:43:03.686668 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:03.686635    2576 generic.go:356] "Generic (PLEG): container finished" podID="c2124fe0-2969-4497-862e-7ea34b7e487c" containerID="ca14bd51a3de24b9b7018fbe899aec427cd04f6c5367e8e1e8acd0f80f9a38cc" exitCode=0
Jul 24 15:43:03.687088 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:03.686710    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rk57t" event={"ID":"c2124fe0-2969-4497-862e-7ea34b7e487c","Type":"ContainerDied","Data":"ca14bd51a3de24b9b7018fbe899aec427cd04f6c5367e8e1e8acd0f80f9a38cc"}
Jul 24 15:43:03.687088 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:03.687033    2576 scope.go:117] "RemoveContainer" containerID="ca14bd51a3de24b9b7018fbe899aec427cd04f6c5367e8e1e8acd0f80f9a38cc"
Jul 24 15:43:04.692681 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:04.692648    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-rk57t" event={"ID":"c2124fe0-2969-4497-862e-7ea34b7e487c","Type":"ContainerStarted","Data":"558632b79053bffb04e4711b0f14675a3fcf34ac4ffd385e0a97139ce0e42afc"}
Jul 24 15:43:51.651636 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.651593    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dd4b6cdc4-2n2lp"]
Jul 24 15:43:51.652230 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.651872    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d54a2a-55ae-4f11-99db-3f98a78db125" containerName="registry"
Jul 24 15:43:51.652230 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.651882    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d54a2a-55ae-4f11-99db-3f98a78db125" containerName="registry"
Jul 24 15:43:51.652230 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.651932    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d54a2a-55ae-4f11-99db-3f98a78db125" containerName="registry"
Jul 24 15:43:51.654985 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.654918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.671600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.671576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd4b6cdc4-2n2lp"]
Jul 24 15:43:51.728192 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.728159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-service-ca\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.728192 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.728190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-trusted-ca-bundle\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.728380 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.728207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4g7\" (UniqueName: \"kubernetes.io/projected/b89d43d6-eb11-419b-a1c2-145818f7c660-kube-api-access-gg4g7\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.728380 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.728230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-oauth-serving-cert\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.728380 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.728314    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-console-config\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.728380 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.728364    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-serving-cert\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.728511 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.728395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-oauth-config\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.828996 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.828968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-serving-cert\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.829151 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.829012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-oauth-config\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.829297 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.829278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-service-ca\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.829339 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.829309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-trusted-ca-bundle\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.829339 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.829326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4g7\" (UniqueName: \"kubernetes.io/projected/b89d43d6-eb11-419b-a1c2-145818f7c660-kube-api-access-gg4g7\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.829424 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.829351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-oauth-serving-cert\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.829424 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.829401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-console-config\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.830098 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.830071    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-service-ca\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.830195 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.830104    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-oauth-serving-cert\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.830195 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.830150    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-trusted-ca-bundle\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.830334 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.830314    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-console-config\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.831697 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.831677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-oauth-config\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.831795 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.831776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-serving-cert\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.838461 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.838439    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4g7\" (UniqueName: \"kubernetes.io/projected/b89d43d6-eb11-419b-a1c2-145818f7c660-kube-api-access-gg4g7\") pod \"console-dd4b6cdc4-2n2lp\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") " pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:51.964657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:51.964570    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:43:52.113016 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:52.112992    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd4b6cdc4-2n2lp"]
Jul 24 15:43:52.115515 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:43:52.115488    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89d43d6_eb11_419b_a1c2_145818f7c660.slice/crio-3cd00a33afe586eeacb42371642e43763deb4e627ed44cb1188ac0191d84e087 WatchSource:0}: Error finding container 3cd00a33afe586eeacb42371642e43763deb4e627ed44cb1188ac0191d84e087: Status 404 returned error can't find the container with id 3cd00a33afe586eeacb42371642e43763deb4e627ed44cb1188ac0191d84e087
Jul 24 15:43:52.870218 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:52.870184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd4b6cdc4-2n2lp" event={"ID":"b89d43d6-eb11-419b-a1c2-145818f7c660","Type":"ContainerStarted","Data":"b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c"}
Jul 24 15:43:52.870617 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:52.870253    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd4b6cdc4-2n2lp" event={"ID":"b89d43d6-eb11-419b-a1c2-145818f7c660","Type":"ContainerStarted","Data":"3cd00a33afe586eeacb42371642e43763deb4e627ed44cb1188ac0191d84e087"}
Jul 24 15:43:52.889956 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:43:52.889893    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dd4b6cdc4-2n2lp" podStartSLOduration=1.889879139 podStartE2EDuration="1.889879139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:43:52.888519632 +0000 UTC m=+192.373080230" watchObservedRunningTime="2026-07-24 15:43:52.889879139 +0000 UTC m=+192.374439737"
Jul 24 15:44:01.964828 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:01.964794    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:44:01.965433 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:01.964967    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:44:01.969888 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:01.969865    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:44:02.910387 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:02.910360    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:44:02.957694 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:02.957662    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66965b8bcb-wllnh"]
Jul 24 15:44:27.981158 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:27.981102    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66965b8bcb-wllnh" podUID="d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" containerName="console" containerID="cri-o://808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817" gracePeriod=15
Jul 24 15:44:28.227292 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.227259    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:44:28.318733 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.318651    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-trusted-ca-bundle\") pod \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") "
Jul 24 15:44:28.318733 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.318733    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-config\") pod \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") "
Jul 24 15:44:28.318937 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.318761    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-oauth-config\") pod \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") "
Jul 24 15:44:28.318937 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.318781    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-serving-cert\") pod \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") "
Jul 24 15:44:28.318937 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.318810    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66wf6\" (UniqueName: \"kubernetes.io/projected/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-kube-api-access-66wf6\") pod \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") "
Jul 24 15:44:28.318937 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.318836    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-service-ca\") pod \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") "
Jul 24 15:44:28.318937 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.318864    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-oauth-serving-cert\") pod \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\" (UID: \"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458\") "
Jul 24 15:44:28.319230 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.319081    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" (UID: "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:44:28.319230 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.319134    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-config" (OuterVolumeSpecName: "console-config") pod "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" (UID: "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:44:28.319404 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.319377    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-service-ca" (OuterVolumeSpecName: "service-ca") pod "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" (UID: "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:44:28.319458 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.319381    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" (UID: "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:44:28.321326 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.321296    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" (UID: "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:44:28.321545 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.321518    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" (UID: "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:44:28.321545 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.321537    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-kube-api-access-66wf6" (OuterVolumeSpecName: "kube-api-access-66wf6") pod "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" (UID: "d186fad1-0cf3-43b1-9d7d-e51cf5ea0458"). InnerVolumeSpecName "kube-api-access-66wf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:44:28.419465 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.419428    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:44:28.419465 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.419457    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-oauth-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:44:28.419465 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.419468    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-console-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:44:28.419465 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.419477    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wf6\" (UniqueName: \"kubernetes.io/projected/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-kube-api-access-66wf6\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:44:28.419722 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.419485    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:44:28.419722 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.419496    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-oauth-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:44:28.419722 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:28.419510    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458-trusted-ca-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:44:29.008134 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:29.008101    2576 generic.go:356] "Generic (PLEG): container finished" podID="d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" containerID="808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817" exitCode=0
Jul 24 15:44:29.008134 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:29.008138    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66965b8bcb-wllnh"
Jul 24 15:44:29.008646 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:29.008156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66965b8bcb-wllnh" event={"ID":"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458","Type":"ContainerDied","Data":"808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817"}
Jul 24 15:44:29.008646 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:29.008199    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66965b8bcb-wllnh" event={"ID":"d186fad1-0cf3-43b1-9d7d-e51cf5ea0458","Type":"ContainerDied","Data":"152a567b9aefd7b8dad6c04029ae8f46d4d206692f593ea2eb2309fde4597e53"}
Jul 24 15:44:29.008646 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:29.008226    2576 scope.go:117] "RemoveContainer" containerID="808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817"
Jul 24 15:44:29.017225 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:29.017206    2576 scope.go:117] "RemoveContainer" containerID="808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817"
Jul 24 15:44:29.017513 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:44:29.017494    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817\": container with ID starting with 808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817 not found: ID does not exist" containerID="808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817"
Jul 24 15:44:29.017564 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:29.017522    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817"} err="failed to get container status \"808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817\": rpc error: code = NotFound desc = could not find container \"808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817\": container with ID starting with 808211ad86fa08d7169ca8818750d88ced9dcdc8f6f1fcef8042b6d1e2592817 not found: ID does not exist"
Jul 24 15:44:29.029465 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:29.029440    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66965b8bcb-wllnh"]
Jul 24 15:44:29.032098 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:29.032077    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66965b8bcb-wllnh"]
Jul 24 15:44:31.004669 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:44:31.004629    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" path="/var/lib/kubelet/pods/d186fad1-0cf3-43b1-9d7d-e51cf5ea0458/volumes"
Jul 24 15:45:07.010771 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.010738    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fd57fc997-lhhch"]
Jul 24 15:45:07.011366 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.011132    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" containerName="console"
Jul 24 15:45:07.011366 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.011150    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" containerName="console"
Jul 24 15:45:07.011366 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.011227    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d186fad1-0cf3-43b1-9d7d-e51cf5ea0458" containerName="console"
Jul 24 15:45:07.014425 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.014401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.027783 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.027757    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd57fc997-lhhch"]
Jul 24 15:45:07.031611 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.031593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-oauth-config\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.031722 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.031623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-trusted-ca-bundle\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.031722 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.031647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-serving-cert\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.031722 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.031662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ww7s\" (UniqueName: \"kubernetes.io/projected/b956f06d-2df7-4010-bf38-44f7c39b2102-kube-api-access-6ww7s\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.031722 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.031681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-oauth-serving-cert\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.031722 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.031718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-console-config\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.031884 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.031743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-service-ca\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.132705 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.132668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-oauth-config\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.132705 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.132708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-trusted-ca-bundle\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.132961 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.132764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-serving-cert\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.132961 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.132788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ww7s\" (UniqueName: \"kubernetes.io/projected/b956f06d-2df7-4010-bf38-44f7c39b2102-kube-api-access-6ww7s\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.132961 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.132813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-oauth-serving-cert\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.132961 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.132843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-console-config\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.132961 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.132885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-service-ca\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.133627 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.133578    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-oauth-serving-cert\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.133755 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.133735    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-trusted-ca-bundle\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.133919 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.133896    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-console-config\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.133978 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.133953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-service-ca\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.135570 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.135540    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-oauth-config\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.135666 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.135629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-serving-cert\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.140742 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.140722    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ww7s\" (UniqueName: \"kubernetes.io/projected/b956f06d-2df7-4010-bf38-44f7c39b2102-kube-api-access-6ww7s\") pod \"console-6fd57fc997-lhhch\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") " pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.327396 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.327312    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:07.456791 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:07.456766    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd57fc997-lhhch"]
Jul 24 15:45:07.458847 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:45:07.458819    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb956f06d_2df7_4010_bf38_44f7c39b2102.slice/crio-847914e2e6aade03171df349b6d3ac7cd411c27c51906b3dfcf17c4b10511587 WatchSource:0}: Error finding container 847914e2e6aade03171df349b6d3ac7cd411c27c51906b3dfcf17c4b10511587: Status 404 returned error can't find the container with id 847914e2e6aade03171df349b6d3ac7cd411c27c51906b3dfcf17c4b10511587
Jul 24 15:45:08.150115 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:08.150084    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd57fc997-lhhch" event={"ID":"b956f06d-2df7-4010-bf38-44f7c39b2102","Type":"ContainerStarted","Data":"1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b"}
Jul 24 15:45:08.150521 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:08.150132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd57fc997-lhhch" event={"ID":"b956f06d-2df7-4010-bf38-44f7c39b2102","Type":"ContainerStarted","Data":"847914e2e6aade03171df349b6d3ac7cd411c27c51906b3dfcf17c4b10511587"}
Jul 24 15:45:08.168200 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:08.168059    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fd57fc997-lhhch" podStartSLOduration=2.168040795 podStartE2EDuration="2.168040795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:08.167907913 +0000 UTC m=+267.652468512" watchObservedRunningTime="2026-07-24 15:45:08.168040795 +0000 UTC m=+267.652601393"
Jul 24 15:45:17.327963 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:17.327924    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:17.328478 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:17.327977    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:17.332625 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:17.332603    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:18.192797 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:18.192764    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:45:18.240598 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:18.240559    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd4b6cdc4-2n2lp"]
Jul 24 15:45:40.932233 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:40.932206    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 15:45:40.937788 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:40.937766    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:45:40.944208 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:40.944184    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 15:45:40.944921 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:40.944896    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 15:45:40.950150 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:40.950132    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:45:42.010775 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.010744    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"]
Jul 24 15:45:42.013923 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.013907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.016481 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.016460    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 15:45:42.016481 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.016460    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 15:45:42.016648 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.016462    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8t8cg\""
Jul 24 15:45:42.024296 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.024278    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"]
Jul 24 15:45:42.106505 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.106479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.106673 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.106524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.106673 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.106567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt84j\" (UniqueName: \"kubernetes.io/projected/75a02be0-5bb7-42c1-b17c-06d96fbb230e-kube-api-access-lt84j\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.207548 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.207521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.207707 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.207578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.207707 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.207611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt84j\" (UniqueName: \"kubernetes.io/projected/75a02be0-5bb7-42c1-b17c-06d96fbb230e-kube-api-access-lt84j\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.207941 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.207918    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.208004 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.207938    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.215201 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.215181    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt84j\" (UniqueName: \"kubernetes.io/projected/75a02be0-5bb7-42c1-b17c-06d96fbb230e-kube-api-access-lt84j\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.322981 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.322906    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:45:42.452404 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.452364    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"]
Jul 24 15:45:42.455276 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:45:42.455228    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a02be0_5bb7_42c1_b17c_06d96fbb230e.slice/crio-fb688b2f23da4cf9791a30c311954dc3c28498c3ffe7b18b8481a62aa51ff935 WatchSource:0}: Error finding container fb688b2f23da4cf9791a30c311954dc3c28498c3ffe7b18b8481a62aa51ff935: Status 404 returned error can't find the container with id fb688b2f23da4cf9791a30c311954dc3c28498c3ffe7b18b8481a62aa51ff935
Jul 24 15:45:42.457224 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:42.457204    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:45:43.263426 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.263378    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dd4b6cdc4-2n2lp" podUID="b89d43d6-eb11-419b-a1c2-145818f7c660" containerName="console" containerID="cri-o://b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c" gracePeriod=15
Jul 24 15:45:43.282103 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.282070    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj" event={"ID":"75a02be0-5bb7-42c1-b17c-06d96fbb230e","Type":"ContainerStarted","Data":"fb688b2f23da4cf9791a30c311954dc3c28498c3ffe7b18b8481a62aa51ff935"}
Jul 24 15:45:43.527639 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.527587    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:45:43.619529 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.619485    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-console-config\") pod \"b89d43d6-eb11-419b-a1c2-145818f7c660\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") "
Jul 24 15:45:43.619529 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.619526    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-service-ca\") pod \"b89d43d6-eb11-419b-a1c2-145818f7c660\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") "
Jul 24 15:45:43.619811 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.619558    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-oauth-serving-cert\") pod \"b89d43d6-eb11-419b-a1c2-145818f7c660\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") "
Jul 24 15:45:43.619811 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.619585    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-serving-cert\") pod \"b89d43d6-eb11-419b-a1c2-145818f7c660\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") "
Jul 24 15:45:43.619811 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.619631    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-oauth-config\") pod \"b89d43d6-eb11-419b-a1c2-145818f7c660\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") "
Jul 24 15:45:43.619811 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.619671    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg4g7\" (UniqueName: \"kubernetes.io/projected/b89d43d6-eb11-419b-a1c2-145818f7c660-kube-api-access-gg4g7\") pod \"b89d43d6-eb11-419b-a1c2-145818f7c660\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") "
Jul 24 15:45:43.619811 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.619709    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-trusted-ca-bundle\") pod \"b89d43d6-eb11-419b-a1c2-145818f7c660\" (UID: \"b89d43d6-eb11-419b-a1c2-145818f7c660\") "
Jul 24 15:45:43.620056 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.619917    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-service-ca" (OuterVolumeSpecName: "service-ca") pod "b89d43d6-eb11-419b-a1c2-145818f7c660" (UID: "b89d43d6-eb11-419b-a1c2-145818f7c660"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:45:43.620056 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.619977    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b89d43d6-eb11-419b-a1c2-145818f7c660" (UID: "b89d43d6-eb11-419b-a1c2-145818f7c660"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:45:43.620307 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.620279    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b89d43d6-eb11-419b-a1c2-145818f7c660" (UID: "b89d43d6-eb11-419b-a1c2-145818f7c660"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:45:43.620436 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.620314    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-console-config" (OuterVolumeSpecName: "console-config") pod "b89d43d6-eb11-419b-a1c2-145818f7c660" (UID: "b89d43d6-eb11-419b-a1c2-145818f7c660"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:45:43.622433 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.622404    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89d43d6-eb11-419b-a1c2-145818f7c660-kube-api-access-gg4g7" (OuterVolumeSpecName: "kube-api-access-gg4g7") pod "b89d43d6-eb11-419b-a1c2-145818f7c660" (UID: "b89d43d6-eb11-419b-a1c2-145818f7c660"). InnerVolumeSpecName "kube-api-access-gg4g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:45:43.622433 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.622426    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b89d43d6-eb11-419b-a1c2-145818f7c660" (UID: "b89d43d6-eb11-419b-a1c2-145818f7c660"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:45:43.622577 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.622444    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b89d43d6-eb11-419b-a1c2-145818f7c660" (UID: "b89d43d6-eb11-419b-a1c2-145818f7c660"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:45:43.721033 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.721003    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4g7\" (UniqueName: \"kubernetes.io/projected/b89d43d6-eb11-419b-a1c2-145818f7c660-kube-api-access-gg4g7\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:45:43.721033 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.721030    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-trusted-ca-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:45:43.721186 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.721040    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-console-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:45:43.721186 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.721049    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:45:43.721186 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.721057    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b89d43d6-eb11-419b-a1c2-145818f7c660-oauth-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:45:43.721186 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.721068    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:45:43.721186 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:43.721080    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b89d43d6-eb11-419b-a1c2-145818f7c660-console-oauth-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:45:44.288157 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:44.288113    2576 generic.go:356] "Generic (PLEG): container finished" podID="b89d43d6-eb11-419b-a1c2-145818f7c660" containerID="b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c" exitCode=0
Jul 24 15:45:44.288157 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:44.288151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd4b6cdc4-2n2lp" event={"ID":"b89d43d6-eb11-419b-a1c2-145818f7c660","Type":"ContainerDied","Data":"b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c"}
Jul 24 15:45:44.288705 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:44.288179    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd4b6cdc4-2n2lp"
Jul 24 15:45:44.288705 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:44.288214    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd4b6cdc4-2n2lp" event={"ID":"b89d43d6-eb11-419b-a1c2-145818f7c660","Type":"ContainerDied","Data":"3cd00a33afe586eeacb42371642e43763deb4e627ed44cb1188ac0191d84e087"}
Jul 24 15:45:44.288705 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:44.288256    2576 scope.go:117] "RemoveContainer" containerID="b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c"
Jul 24 15:45:44.298982 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:44.298960    2576 scope.go:117] "RemoveContainer" containerID="b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c"
Jul 24 15:45:44.299465 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:45:44.299436    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c\": container with ID starting with b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c not found: ID does not exist" containerID="b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c"
Jul 24 15:45:44.299565 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:44.299478    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c"} err="failed to get container status \"b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c\": rpc error: code = NotFound desc = could not find container \"b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c\": container with ID starting with b9549419a6f68e87afdc72f830da0ee026e23507eb48ab5b32e3f34bd38d799c not found: ID does not exist"
Jul 24 15:45:44.314153 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:44.314120    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd4b6cdc4-2n2lp"]
Jul 24 15:45:44.318352 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:44.318311    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dd4b6cdc4-2n2lp"]
Jul 24 15:45:45.005321 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:45.005285    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b89d43d6-eb11-419b-a1c2-145818f7c660" path="/var/lib/kubelet/pods/b89d43d6-eb11-419b-a1c2-145818f7c660/volumes"
Jul 24 15:45:49.315004 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:49.314969    2576 generic.go:356] "Generic (PLEG): container finished" podID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerID="c2f93455643355ab5e544254a1f43586ed6de0f0251db9d915e0474b2f3b96a1" exitCode=0
Jul 24 15:45:49.315377 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:49.315054    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj" event={"ID":"75a02be0-5bb7-42c1-b17c-06d96fbb230e","Type":"ContainerDied","Data":"c2f93455643355ab5e544254a1f43586ed6de0f0251db9d915e0474b2f3b96a1"}
Jul 24 15:45:52.328630 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:52.328593    2576 generic.go:356] "Generic (PLEG): container finished" podID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerID="6fe8320d16a23c3926288a4304627dfad560014e8daec47a3e2091c357ba5343" exitCode=0
Jul 24 15:45:52.329036 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:45:52.328692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj" event={"ID":"75a02be0-5bb7-42c1-b17c-06d96fbb230e","Type":"ContainerDied","Data":"6fe8320d16a23c3926288a4304627dfad560014e8daec47a3e2091c357ba5343"}
Jul 24 15:46:00.365331 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:00.365291    2576 generic.go:356] "Generic (PLEG): container finished" podID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerID="18c8584ffcf4e844af08cc41aae9835227aa4e95dcec0ce0f6f81b7903c1d9ff" exitCode=0
Jul 24 15:46:00.365689 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:00.365384    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj" event={"ID":"75a02be0-5bb7-42c1-b17c-06d96fbb230e","Type":"ContainerDied","Data":"18c8584ffcf4e844af08cc41aae9835227aa4e95dcec0ce0f6f81b7903c1d9ff"}
Jul 24 15:46:01.506848 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.506820    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:46:01.555091 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.555051    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt84j\" (UniqueName: \"kubernetes.io/projected/75a02be0-5bb7-42c1-b17c-06d96fbb230e-kube-api-access-lt84j\") pod \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") "
Jul 24 15:46:01.555091 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.555094    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-bundle\") pod \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") "
Jul 24 15:46:01.555381 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.555116    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-util\") pod \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\" (UID: \"75a02be0-5bb7-42c1-b17c-06d96fbb230e\") "
Jul 24 15:46:01.555719 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.555694    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-bundle" (OuterVolumeSpecName: "bundle") pod "75a02be0-5bb7-42c1-b17c-06d96fbb230e" (UID: "75a02be0-5bb7-42c1-b17c-06d96fbb230e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:01.557463 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.557436    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a02be0-5bb7-42c1-b17c-06d96fbb230e-kube-api-access-lt84j" (OuterVolumeSpecName: "kube-api-access-lt84j") pod "75a02be0-5bb7-42c1-b17c-06d96fbb230e" (UID: "75a02be0-5bb7-42c1-b17c-06d96fbb230e"). InnerVolumeSpecName "kube-api-access-lt84j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:01.560223 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.560194    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-util" (OuterVolumeSpecName: "util") pod "75a02be0-5bb7-42c1-b17c-06d96fbb230e" (UID: "75a02be0-5bb7-42c1-b17c-06d96fbb230e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:01.655604 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.655560    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-util\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:46:01.655604 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.655603    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt84j\" (UniqueName: \"kubernetes.io/projected/75a02be0-5bb7-42c1-b17c-06d96fbb230e-kube-api-access-lt84j\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:46:01.655604 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:01.655616    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75a02be0-5bb7-42c1-b17c-06d96fbb230e-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:46:02.374472 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:02.374443    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj"
Jul 24 15:46:02.374651 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:02.374437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mmmbj" event={"ID":"75a02be0-5bb7-42c1-b17c-06d96fbb230e","Type":"ContainerDied","Data":"fb688b2f23da4cf9791a30c311954dc3c28498c3ffe7b18b8481a62aa51ff935"}
Jul 24 15:46:02.374651 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:02.374565    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb688b2f23da4cf9791a30c311954dc3c28498c3ffe7b18b8481a62aa51ff935"
Jul 24 15:46:10.010966 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.010925    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"]
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011337    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89d43d6-eb11-419b-a1c2-145818f7c660" containerName="console"
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011354    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89d43d6-eb11-419b-a1c2-145818f7c660" containerName="console"
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011369    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerName="pull"
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011377    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerName="pull"
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011389    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerName="util"
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011396    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerName="util"
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011413    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerName="extract"
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011421    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerName="extract"
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011492    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89d43d6-eb11-419b-a1c2-145818f7c660" containerName="console"
Jul 24 15:46:10.011567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.011507    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a02be0-5bb7-42c1-b17c-06d96fbb230e" containerName="extract"
Jul 24 15:46:10.015799 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.015776    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"
Jul 24 15:46:10.018402 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.018381    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:46:10.018531 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.018412    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 24 15:46:10.018531 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.018423    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-skpmj\""
Jul 24 15:46:10.025929 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.025906    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"]
Jul 24 15:46:10.119451 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.119423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8gc7\" (UID: \"1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"
Jul 24 15:46:10.119635 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.119459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cpmd\" (UniqueName: \"kubernetes.io/projected/1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb-kube-api-access-4cpmd\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8gc7\" (UID: \"1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"
Jul 24 15:46:10.219985 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.219954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8gc7\" (UID: \"1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"
Jul 24 15:46:10.220144 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.219994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cpmd\" (UniqueName: \"kubernetes.io/projected/1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb-kube-api-access-4cpmd\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8gc7\" (UID: \"1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"
Jul 24 15:46:10.220326 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.220309    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8gc7\" (UID: \"1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"
Jul 24 15:46:10.228976 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.228950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cpmd\" (UniqueName: \"kubernetes.io/projected/1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb-kube-api-access-4cpmd\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8gc7\" (UID: \"1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"
Jul 24 15:46:10.324868 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.324790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"
Jul 24 15:46:10.464257 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:10.464211    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7"]
Jul 24 15:46:10.466208 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:46:10.466177    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1094a7c4_4bc1_42ee_b7d2_a6bf6c5fddbb.slice/crio-f5f8730d860ecbeb5a31dfa6ac6c58c07297fad15e9d9eb84a70a45747137cd7 WatchSource:0}: Error finding container f5f8730d860ecbeb5a31dfa6ac6c58c07297fad15e9d9eb84a70a45747137cd7: Status 404 returned error can't find the container with id f5f8730d860ecbeb5a31dfa6ac6c58c07297fad15e9d9eb84a70a45747137cd7
Jul 24 15:46:11.411040 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:11.411006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7" event={"ID":"1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb","Type":"ContainerStarted","Data":"f5f8730d860ecbeb5a31dfa6ac6c58c07297fad15e9d9eb84a70a45747137cd7"}
Jul 24 15:46:25.471561 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:25.471528    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7" event={"ID":"1094a7c4-4bc1-42ee-b7d2-a6bf6c5fddbb","Type":"ContainerStarted","Data":"3d0facd6590d2c53a9ea3a82672b25a68c22f9be9f808d024ea0d9be11a142ba"}
Jul 24 15:46:25.495994 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:25.495936    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8gc7" podStartSLOduration=1.83976002 podStartE2EDuration="16.495918716s" totalImagesPullingTime="14.656158696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:25.493859744 +0000 UTC m=+344.978420343" watchObservedRunningTime="2026-07-24 15:46:25.495918716 +0000 UTC m=+344.980479314"
Jul 24 15:46:26.192521 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.192471    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"]
Jul 24 15:46:26.194785 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.194761    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.197274 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.197253    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8t8cg\""
Jul 24 15:46:26.197625 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.197612    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 15:46:26.198198 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.198181    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 15:46:26.210879 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.210851    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"]
Jul 24 15:46:26.256094 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.256059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.256276 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.256119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/d0fa769d-81b7-430c-84cb-67f328cc249c-kube-api-access-dvbz4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.256276 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.256177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.356652 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.356624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/d0fa769d-81b7-430c-84cb-67f328cc249c-kube-api-access-dvbz4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.356827 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.356666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.356827 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.356709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.357133 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.357113    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.357180 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.357131    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.367139 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.367110    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/d0fa769d-81b7-430c-84cb-67f328cc249c-kube-api-access-dvbz4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.503773 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.503690    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:26.640707 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:26.640683    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"]
Jul 24 15:46:26.643095 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:46:26.643070    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fa769d_81b7_430c_84cb_67f328cc249c.slice/crio-9bfff4e5181f75c426150be6d1d71a33a752cf7b7e4ee0b2d013855b3b138c7b WatchSource:0}: Error finding container 9bfff4e5181f75c426150be6d1d71a33a752cf7b7e4ee0b2d013855b3b138c7b: Status 404 returned error can't find the container with id 9bfff4e5181f75c426150be6d1d71a33a752cf7b7e4ee0b2d013855b3b138c7b
Jul 24 15:46:27.481008 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:27.480975    2576 generic.go:356] "Generic (PLEG): container finished" podID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerID="7ccfa1d8e2d30d184f7f03bdf3a4dbf1f9375df9923a3edcc60cbf3d4dd4107a" exitCode=0
Jul 24 15:46:27.481198 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:27.481055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh" event={"ID":"d0fa769d-81b7-430c-84cb-67f328cc249c","Type":"ContainerDied","Data":"7ccfa1d8e2d30d184f7f03bdf3a4dbf1f9375df9923a3edcc60cbf3d4dd4107a"}
Jul 24 15:46:27.481198 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:27.481104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh" event={"ID":"d0fa769d-81b7-430c-84cb-67f328cc249c","Type":"ContainerStarted","Data":"9bfff4e5181f75c426150be6d1d71a33a752cf7b7e4ee0b2d013855b3b138c7b"}
Jul 24 15:46:30.496049 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:30.496010    2576 generic.go:356] "Generic (PLEG): container finished" podID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerID="7caf0e3f70df87ad99185ecda7aac85263ccc4c528e476bbfba777049bfe6d53" exitCode=0
Jul 24 15:46:30.496502 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:30.496061    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh" event={"ID":"d0fa769d-81b7-430c-84cb-67f328cc249c","Type":"ContainerDied","Data":"7caf0e3f70df87ad99185ecda7aac85263ccc4c528e476bbfba777049bfe6d53"}
Jul 24 15:46:31.502630 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:31.502599    2576 generic.go:356] "Generic (PLEG): container finished" podID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerID="2e16559b83e62345f6e74a0630735cf6701d7c033e8a1865f3c1c692dcf22a2f" exitCode=0
Jul 24 15:46:31.503020 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:31.502649    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh" event={"ID":"d0fa769d-81b7-430c-84cb-67f328cc249c","Type":"ContainerDied","Data":"2e16559b83e62345f6e74a0630735cf6701d7c033e8a1865f3c1c692dcf22a2f"}
Jul 24 15:46:32.637779 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.637757    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:32.714013 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.713981    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/d0fa769d-81b7-430c-84cb-67f328cc249c-kube-api-access-dvbz4\") pod \"d0fa769d-81b7-430c-84cb-67f328cc249c\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") "
Jul 24 15:46:32.714167 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.714037    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-bundle\") pod \"d0fa769d-81b7-430c-84cb-67f328cc249c\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") "
Jul 24 15:46:32.714167 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.714062    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-util\") pod \"d0fa769d-81b7-430c-84cb-67f328cc249c\" (UID: \"d0fa769d-81b7-430c-84cb-67f328cc249c\") "
Jul 24 15:46:32.714454 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.714423    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-bundle" (OuterVolumeSpecName: "bundle") pod "d0fa769d-81b7-430c-84cb-67f328cc249c" (UID: "d0fa769d-81b7-430c-84cb-67f328cc249c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:32.716187 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.716159    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fa769d-81b7-430c-84cb-67f328cc249c-kube-api-access-dvbz4" (OuterVolumeSpecName: "kube-api-access-dvbz4") pod "d0fa769d-81b7-430c-84cb-67f328cc249c" (UID: "d0fa769d-81b7-430c-84cb-67f328cc249c"). InnerVolumeSpecName "kube-api-access-dvbz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:32.719059 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.719015    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-util" (OuterVolumeSpecName: "util") pod "d0fa769d-81b7-430c-84cb-67f328cc249c" (UID: "d0fa769d-81b7-430c-84cb-67f328cc249c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:32.814739 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.814673    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-util\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:46:32.814739 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.814703    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbz4\" (UniqueName: \"kubernetes.io/projected/d0fa769d-81b7-430c-84cb-67f328cc249c-kube-api-access-dvbz4\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:46:32.814739 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:32.814712    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0fa769d-81b7-430c-84cb-67f328cc249c-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:46:33.519979 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:33.519948    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh"
Jul 24 15:46:33.519979 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:33.519954    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvnsbh" event={"ID":"d0fa769d-81b7-430c-84cb-67f328cc249c","Type":"ContainerDied","Data":"9bfff4e5181f75c426150be6d1d71a33a752cf7b7e4ee0b2d013855b3b138c7b"}
Jul 24 15:46:33.520226 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:33.520019    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bfff4e5181f75c426150be6d1d71a33a752cf7b7e4ee0b2d013855b3b138c7b"
Jul 24 15:46:34.479773 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.479739    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dkwxf"]
Jul 24 15:46:34.480169 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.480041    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerName="util"
Jul 24 15:46:34.480169 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.480052    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerName="util"
Jul 24 15:46:34.480169 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.480066    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerName="pull"
Jul 24 15:46:34.480169 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.480071    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerName="pull"
Jul 24 15:46:34.480169 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.480079    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerName="extract"
Jul 24 15:46:34.480169 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.480084    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerName="extract"
Jul 24 15:46:34.480169 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.480134    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fa769d-81b7-430c-84cb-67f328cc249c" containerName="extract"
Jul 24 15:46:34.506522 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.506498    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dkwxf"]
Jul 24 15:46:34.506711 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.506581    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:34.509229 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.509205    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 15:46:34.510278 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.510259    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 15:46:34.510391 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.510309    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-smjqk\""
Jul 24 15:46:34.628630 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.628593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpdl\" (UniqueName: \"kubernetes.io/projected/f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8-kube-api-access-wmpdl\") pod \"cert-manager-webhook-6486ffd997-dkwxf\" (UID: \"f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:34.628810 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.628647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dkwxf\" (UID: \"f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:34.729441 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.729404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpdl\" (UniqueName: \"kubernetes.io/projected/f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8-kube-api-access-wmpdl\") pod \"cert-manager-webhook-6486ffd997-dkwxf\" (UID: \"f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:34.729597 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.729457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dkwxf\" (UID: \"f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:34.738187 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.738112    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dkwxf\" (UID: \"f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:34.738332 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.738187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpdl\" (UniqueName: \"kubernetes.io/projected/f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8-kube-api-access-wmpdl\") pod \"cert-manager-webhook-6486ffd997-dkwxf\" (UID: \"f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:34.822198 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.822168    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:34.964975 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:34.964929    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dkwxf"]
Jul 24 15:46:34.967005 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:46:34.966974    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2a9bddd_b3e2_4b75_8e6f_a560cd1a49d8.slice/crio-185de0145a7f0e23f2e2e24dc0dd972827d7ff8787fcf02ced615d27014f4f4a WatchSource:0}: Error finding container 185de0145a7f0e23f2e2e24dc0dd972827d7ff8787fcf02ced615d27014f4f4a: Status 404 returned error can't find the container with id 185de0145a7f0e23f2e2e24dc0dd972827d7ff8787fcf02ced615d27014f4f4a
Jul 24 15:46:35.530420 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:35.530380    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf" event={"ID":"f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8","Type":"ContainerStarted","Data":"185de0145a7f0e23f2e2e24dc0dd972827d7ff8787fcf02ced615d27014f4f4a"}
Jul 24 15:46:37.540719 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:37.540682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf" event={"ID":"f2a9bddd-b3e2-4b75-8e6f-a560cd1a49d8","Type":"ContainerStarted","Data":"f5e1a076de72792c38eb5d39e0eb4d8a9c058df878fcac5e1a4b19de9663a4dc"}
Jul 24 15:46:37.541123 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:37.540801    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:37.555767 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:37.555722    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf" podStartSLOduration=1.136024155 podStartE2EDuration="3.555707391s" totalImagesPullingTime="2.419683236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:37.555660442 +0000 UTC m=+357.040221065" watchObservedRunningTime="2026-07-24 15:46:37.555707391 +0000 UTC m=+357.040267989"
Jul 24 15:46:43.547530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:43.547497    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dkwxf"
Jul 24 15:46:44.757851 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:44.757815    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"]
Jul 24 15:46:44.760840 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:44.760818    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:44.763464 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:44.763440    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8t8cg\""
Jul 24 15:46:44.763584 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:44.763452    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 15:46:44.764310 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:44.764287    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 15:46:44.769209 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:44.769186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"]
Jul 24 15:46:44.915864 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:44.915832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:44.916020 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:44.915869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwdq\" (UniqueName: \"kubernetes.io/projected/bfecbab7-8007-406a-aeab-7b30611ea206-kube-api-access-tfwdq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:44.916020 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:44.915959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:45.019338 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.019236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:45.019496 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.019337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:45.019496 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.019376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwdq\" (UniqueName: \"kubernetes.io/projected/bfecbab7-8007-406a-aeab-7b30611ea206-kube-api-access-tfwdq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:45.019709 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.019684    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:45.019774 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.019714    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:45.028423 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.028395    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwdq\" (UniqueName: \"kubernetes.io/projected/bfecbab7-8007-406a-aeab-7b30611ea206-kube-api-access-tfwdq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:45.070725 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.070701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:45.201274 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.201229    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"]
Jul 24 15:46:45.203455 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:46:45.203429    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfecbab7_8007_406a_aeab_7b30611ea206.slice/crio-bdd5adb6e4bb23cbc1663a696cff2125207891898900b57d5d681946a32774db WatchSource:0}: Error finding container bdd5adb6e4bb23cbc1663a696cff2125207891898900b57d5d681946a32774db: Status 404 returned error can't find the container with id bdd5adb6e4bb23cbc1663a696cff2125207891898900b57d5d681946a32774db
Jul 24 15:46:45.575258 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.575142    2576 generic.go:356] "Generic (PLEG): container finished" podID="bfecbab7-8007-406a-aeab-7b30611ea206" containerID="cbc3d6b9d0554b8f123968df06ca385bf5d18fe097e250a1efa1781959481a0c" exitCode=0
Jul 24 15:46:45.575417 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.575233    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw" event={"ID":"bfecbab7-8007-406a-aeab-7b30611ea206","Type":"ContainerDied","Data":"cbc3d6b9d0554b8f123968df06ca385bf5d18fe097e250a1efa1781959481a0c"}
Jul 24 15:46:45.575417 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:45.575312    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw" event={"ID":"bfecbab7-8007-406a-aeab-7b30611ea206","Type":"ContainerStarted","Data":"bdd5adb6e4bb23cbc1663a696cff2125207891898900b57d5d681946a32774db"}
Jul 24 15:46:46.582177 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:46.582143    2576 generic.go:356] "Generic (PLEG): container finished" podID="bfecbab7-8007-406a-aeab-7b30611ea206" containerID="d37b76238b6d0e6882b7a0ae2775488efda0feaf10d3e352ae58a7205dde5cc7" exitCode=0
Jul 24 15:46:46.582577 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:46.582187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw" event={"ID":"bfecbab7-8007-406a-aeab-7b30611ea206","Type":"ContainerDied","Data":"d37b76238b6d0e6882b7a0ae2775488efda0feaf10d3e352ae58a7205dde5cc7"}
Jul 24 15:46:47.588872 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:47.588833    2576 generic.go:356] "Generic (PLEG): container finished" podID="bfecbab7-8007-406a-aeab-7b30611ea206" containerID="9b8e86cb2fd7148793a891c9ce26995d8de64e67c2e7ab8388625865022b3414" exitCode=0
Jul 24 15:46:47.589289 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:47.588882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw" event={"ID":"bfecbab7-8007-406a-aeab-7b30611ea206","Type":"ContainerDied","Data":"9b8e86cb2fd7148793a891c9ce26995d8de64e67c2e7ab8388625865022b3414"}
Jul 24 15:46:48.724465 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.724442    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:48.857351 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.857259    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-util\") pod \"bfecbab7-8007-406a-aeab-7b30611ea206\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") "
Jul 24 15:46:48.857351 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.857308    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-bundle\") pod \"bfecbab7-8007-406a-aeab-7b30611ea206\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") "
Jul 24 15:46:48.857351 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.857346    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfwdq\" (UniqueName: \"kubernetes.io/projected/bfecbab7-8007-406a-aeab-7b30611ea206-kube-api-access-tfwdq\") pod \"bfecbab7-8007-406a-aeab-7b30611ea206\" (UID: \"bfecbab7-8007-406a-aeab-7b30611ea206\") "
Jul 24 15:46:48.857989 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.857966    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-bundle" (OuterVolumeSpecName: "bundle") pod "bfecbab7-8007-406a-aeab-7b30611ea206" (UID: "bfecbab7-8007-406a-aeab-7b30611ea206"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:48.859530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.859505    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfecbab7-8007-406a-aeab-7b30611ea206-kube-api-access-tfwdq" (OuterVolumeSpecName: "kube-api-access-tfwdq") pod "bfecbab7-8007-406a-aeab-7b30611ea206" (UID: "bfecbab7-8007-406a-aeab-7b30611ea206"). InnerVolumeSpecName "kube-api-access-tfwdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:46:48.862971 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.862949    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-util" (OuterVolumeSpecName: "util") pod "bfecbab7-8007-406a-aeab-7b30611ea206" (UID: "bfecbab7-8007-406a-aeab-7b30611ea206"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:46:48.958711 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.958676    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-util\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:46:48.958711 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.958707    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfecbab7-8007-406a-aeab-7b30611ea206-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:46:48.958711 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:48.958717    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfwdq\" (UniqueName: \"kubernetes.io/projected/bfecbab7-8007-406a-aeab-7b30611ea206-kube-api-access-tfwdq\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:46:49.599358 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:49.599320    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw" event={"ID":"bfecbab7-8007-406a-aeab-7b30611ea206","Type":"ContainerDied","Data":"bdd5adb6e4bb23cbc1663a696cff2125207891898900b57d5d681946a32774db"}
Jul 24 15:46:49.599358 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:49.599356    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmhzrw"
Jul 24 15:46:49.599567 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:49.599399    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdd5adb6e4bb23cbc1663a696cff2125207891898900b57d5d681946a32774db"
Jul 24 15:46:59.119056 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.119026    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"]
Jul 24 15:46:59.119535 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.119320    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfecbab7-8007-406a-aeab-7b30611ea206" containerName="util"
Jul 24 15:46:59.119535 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.119331    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfecbab7-8007-406a-aeab-7b30611ea206" containerName="util"
Jul 24 15:46:59.119535 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.119339    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfecbab7-8007-406a-aeab-7b30611ea206" containerName="pull"
Jul 24 15:46:59.119535 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.119345    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfecbab7-8007-406a-aeab-7b30611ea206" containerName="pull"
Jul 24 15:46:59.119535 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.119354    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfecbab7-8007-406a-aeab-7b30611ea206" containerName="extract"
Jul 24 15:46:59.119535 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.119359    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfecbab7-8007-406a-aeab-7b30611ea206" containerName="extract"
Jul 24 15:46:59.119535 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.119411    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfecbab7-8007-406a-aeab-7b30611ea206" containerName="extract"
Jul 24 15:46:59.123307 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.123285    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.126068 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.126042    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 15:46:59.126190 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.126059    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rkv9t\""
Jul 24 15:46:59.126262 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.126221    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 15:46:59.126327 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.126311    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 15:46:59.126920 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.126906    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 15:46:59.141379 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.141358    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"]
Jul 24 15:46:59.142522 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.142501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-tmp\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.142626 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.142544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gntw\" (UniqueName: \"kubernetes.io/projected/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-kube-api-access-9gntw\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.142675 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.142624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-webhook-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.142719 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.142676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.142753 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.142744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-manifests\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.243072 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.243030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-manifests\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.243312 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.243078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-tmp\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.243312 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.243201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gntw\" (UniqueName: \"kubernetes.io/projected/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-kube-api-access-9gntw\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.243312 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.243296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-webhook-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.243496 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.243325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.243496 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.243468    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-manifests\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.245693 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.245662    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-tmp\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.245806 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.245760    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-webhook-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.245858 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.245841    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.252879 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.252855    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gntw\" (UniqueName: \"kubernetes.io/projected/b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab-kube-api-access-9gntw\") pod \"opendatahub-operator-controller-manager-6798866cc5-8zmjb\" (UID: \"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.433017 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.432990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:46:59.586759 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.586735    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"]
Jul 24 15:46:59.588730 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:46:59.588698    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f8eb07_ec2a_4cca_9bda_7ecdb1f09fab.slice/crio-ac9a59459739baf2e0d5f759f599e498af560ac678ccd2af50ba508dc04fe91b WatchSource:0}: Error finding container ac9a59459739baf2e0d5f759f599e498af560ac678ccd2af50ba508dc04fe91b: Status 404 returned error can't find the container with id ac9a59459739baf2e0d5f759f599e498af560ac678ccd2af50ba508dc04fe91b
Jul 24 15:46:59.641791 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:46:59.641762    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb" event={"ID":"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab","Type":"ContainerStarted","Data":"ac9a59459739baf2e0d5f759f599e498af560ac678ccd2af50ba508dc04fe91b"}
Jul 24 15:47:02.659480 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:02.659444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb" event={"ID":"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab","Type":"ContainerStarted","Data":"59b59d9be6e8edc9735c0f433eb2d953537441b25afa2783c8c65e2b25e6ff9d"}
Jul 24 15:47:03.665073 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:03.665044    2576 generic.go:356] "Generic (PLEG): container finished" podID="b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab" containerID="59b59d9be6e8edc9735c0f433eb2d953537441b25afa2783c8c65e2b25e6ff9d" exitCode=0
Jul 24 15:47:03.665386 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:03.665121    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb" event={"ID":"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab","Type":"ContainerDied","Data":"59b59d9be6e8edc9735c0f433eb2d953537441b25afa2783c8c65e2b25e6ff9d"}
Jul 24 15:47:04.672015 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:04.671983    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb" event={"ID":"b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab","Type":"ContainerStarted","Data":"95562f5963f9d348c86f3474dfd46f059cc7462a1ae3f79f2d0918d8ca4295e5"}
Jul 24 15:47:04.672441 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:04.672099    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:47:04.694736 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:04.694664    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb" podStartSLOduration=1.349805285 podStartE2EDuration="5.69464789s" totalImagesPullingTime="3.335195733s" totalInitContainerRuntime="1.009646872s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:04.691933569 +0000 UTC m=+384.176494187" watchObservedRunningTime="2026-07-24 15:47:04.69464789 +0000 UTC m=+384.179208487"
Jul 24 15:47:15.679794 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:15.679763    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-8zmjb"
Jul 24 15:47:18.879211 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.879176    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"]
Jul 24 15:47:18.888185 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.888166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:18.891049 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.891029    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 15:47:18.891140 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.891123    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 15:47:18.892088 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.892074    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6m6g4\""
Jul 24 15:47:18.897689 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.897670    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"]
Jul 24 15:47:18.900049 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.900022    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-52rtw"]
Jul 24 15:47:18.903375 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.903355    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:18.906070 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.906051    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 24 15:47:18.906932 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.906914    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2xgd8\""
Jul 24 15:47:18.917861 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:18.917841    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-52rtw"]
Jul 24 15:47:19.006485 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.006453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96fa6894-d9b9-4531-9bd4-a2b3a01c262f-tls-certs\") pod \"kube-auth-proxy-5688cff95c-qjz2g\" (UID: \"96fa6894-d9b9-4531-9bd4-a2b3a01c262f\") " pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.006666 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.006511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2hhv\" (UniqueName: \"kubernetes.io/projected/f0abdb77-eff9-4352-b303-9cc347bef506-kube-api-access-j2hhv\") pod \"odh-model-controller-7998c67dc4-52rtw\" (UID: \"f0abdb77-eff9-4352-b303-9cc347bef506\") " pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:19.006666 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.006534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96fa6894-d9b9-4531-9bd4-a2b3a01c262f-tmp\") pod \"kube-auth-proxy-5688cff95c-qjz2g\" (UID: \"96fa6894-d9b9-4531-9bd4-a2b3a01c262f\") " pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.006666 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.006584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk6zw\" (UniqueName: \"kubernetes.io/projected/96fa6894-d9b9-4531-9bd4-a2b3a01c262f-kube-api-access-tk6zw\") pod \"kube-auth-proxy-5688cff95c-qjz2g\" (UID: \"96fa6894-d9b9-4531-9bd4-a2b3a01c262f\") " pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.006666 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.006653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0abdb77-eff9-4352-b303-9cc347bef506-cert\") pod \"odh-model-controller-7998c67dc4-52rtw\" (UID: \"f0abdb77-eff9-4352-b303-9cc347bef506\") " pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:19.107969 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.107939    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0abdb77-eff9-4352-b303-9cc347bef506-cert\") pod \"odh-model-controller-7998c67dc4-52rtw\" (UID: \"f0abdb77-eff9-4352-b303-9cc347bef506\") " pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:19.108137 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.107991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96fa6894-d9b9-4531-9bd4-a2b3a01c262f-tls-certs\") pod \"kube-auth-proxy-5688cff95c-qjz2g\" (UID: \"96fa6894-d9b9-4531-9bd4-a2b3a01c262f\") " pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.108137 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.108025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2hhv\" (UniqueName: \"kubernetes.io/projected/f0abdb77-eff9-4352-b303-9cc347bef506-kube-api-access-j2hhv\") pod \"odh-model-controller-7998c67dc4-52rtw\" (UID: \"f0abdb77-eff9-4352-b303-9cc347bef506\") " pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:19.108137 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.108041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96fa6894-d9b9-4531-9bd4-a2b3a01c262f-tmp\") pod \"kube-auth-proxy-5688cff95c-qjz2g\" (UID: \"96fa6894-d9b9-4531-9bd4-a2b3a01c262f\") " pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.108137 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.108062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk6zw\" (UniqueName: \"kubernetes.io/projected/96fa6894-d9b9-4531-9bd4-a2b3a01c262f-kube-api-access-tk6zw\") pod \"kube-auth-proxy-5688cff95c-qjz2g\" (UID: \"96fa6894-d9b9-4531-9bd4-a2b3a01c262f\") " pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.108137 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:47:19.108092    2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 24 15:47:19.108411 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:47:19.108182    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0abdb77-eff9-4352-b303-9cc347bef506-cert podName:f0abdb77-eff9-4352-b303-9cc347bef506 nodeName:}" failed. No retries permitted until 2026-07-24 15:47:19.608163427 +0000 UTC m=+399.092724003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f0abdb77-eff9-4352-b303-9cc347bef506-cert") pod "odh-model-controller-7998c67dc4-52rtw" (UID: "f0abdb77-eff9-4352-b303-9cc347bef506") : secret "odh-model-controller-webhook-cert" not found
Jul 24 15:47:19.115051 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.114843    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96fa6894-d9b9-4531-9bd4-a2b3a01c262f-tmp\") pod \"kube-auth-proxy-5688cff95c-qjz2g\" (UID: \"96fa6894-d9b9-4531-9bd4-a2b3a01c262f\") " pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.116465 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.116441    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96fa6894-d9b9-4531-9bd4-a2b3a01c262f-tls-certs\") pod \"kube-auth-proxy-5688cff95c-qjz2g\" (UID: \"96fa6894-d9b9-4531-9bd4-a2b3a01c262f\") " pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.121874 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.121846    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk6zw\" (UniqueName: \"kubernetes.io/projected/96fa6894-d9b9-4531-9bd4-a2b3a01c262f-kube-api-access-tk6zw\") pod \"kube-auth-proxy-5688cff95c-qjz2g\" (UID: \"96fa6894-d9b9-4531-9bd4-a2b3a01c262f\") " pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.122554 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.122533    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2hhv\" (UniqueName: \"kubernetes.io/projected/f0abdb77-eff9-4352-b303-9cc347bef506-kube-api-access-j2hhv\") pod \"odh-model-controller-7998c67dc4-52rtw\" (UID: \"f0abdb77-eff9-4352-b303-9cc347bef506\") " pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:19.197476 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.197453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"
Jul 24 15:47:19.330396 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.330370    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g"]
Jul 24 15:47:19.333220 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:47:19.333192    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fa6894_d9b9_4531_9bd4_a2b3a01c262f.slice/crio-3037b34d3a817064e8932a7c49cf7d6db2aabedc04a675d66a54a209df92dd3d WatchSource:0}: Error finding container 3037b34d3a817064e8932a7c49cf7d6db2aabedc04a675d66a54a209df92dd3d: Status 404 returned error can't find the container with id 3037b34d3a817064e8932a7c49cf7d6db2aabedc04a675d66a54a209df92dd3d
Jul 24 15:47:19.612426 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.612349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0abdb77-eff9-4352-b303-9cc347bef506-cert\") pod \"odh-model-controller-7998c67dc4-52rtw\" (UID: \"f0abdb77-eff9-4352-b303-9cc347bef506\") " pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:19.614874 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.614851    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0abdb77-eff9-4352-b303-9cc347bef506-cert\") pod \"odh-model-controller-7998c67dc4-52rtw\" (UID: \"f0abdb77-eff9-4352-b303-9cc347bef506\") " pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:19.739061 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.739025    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g" event={"ID":"96fa6894-d9b9-4531-9bd4-a2b3a01c262f","Type":"ContainerStarted","Data":"3037b34d3a817064e8932a7c49cf7d6db2aabedc04a675d66a54a209df92dd3d"}
Jul 24 15:47:19.812504 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.812472    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:19.874583 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.874552    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-crbzt"]
Jul 24 15:47:19.879024 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.879000    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:19.881615 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.881592    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hh228\""
Jul 24 15:47:19.881935 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.881616    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 24 15:47:19.889524 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.889496    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-crbzt"]
Jul 24 15:47:19.916019 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.915986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rkkq\" (UniqueName: \"kubernetes.io/projected/da4d0b72-d67a-4eee-b14c-2a18465da204-kube-api-access-8rkkq\") pod \"model-serving-api-54d89bd5fc-crbzt\" (UID: \"da4d0b72-d67a-4eee-b14c-2a18465da204\") " pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:19.916185 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.916049    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4d0b72-d67a-4eee-b14c-2a18465da204-tls-certs\") pod \"model-serving-api-54d89bd5fc-crbzt\" (UID: \"da4d0b72-d67a-4eee-b14c-2a18465da204\") " pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:19.955304 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:19.955272    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-52rtw"]
Jul 24 15:47:19.958413 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:47:19.958392    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0abdb77_eff9_4352_b303_9cc347bef506.slice/crio-fa6b9a2088cf7d88b3325a3e35a8976b52fa1781de952896c519100ab4af1e27 WatchSource:0}: Error finding container fa6b9a2088cf7d88b3325a3e35a8976b52fa1781de952896c519100ab4af1e27: Status 404 returned error can't find the container with id fa6b9a2088cf7d88b3325a3e35a8976b52fa1781de952896c519100ab4af1e27
Jul 24 15:47:20.017124 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.017086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rkkq\" (UniqueName: \"kubernetes.io/projected/da4d0b72-d67a-4eee-b14c-2a18465da204-kube-api-access-8rkkq\") pod \"model-serving-api-54d89bd5fc-crbzt\" (UID: \"da4d0b72-d67a-4eee-b14c-2a18465da204\") " pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:20.017342 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.017138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4d0b72-d67a-4eee-b14c-2a18465da204-tls-certs\") pod \"model-serving-api-54d89bd5fc-crbzt\" (UID: \"da4d0b72-d67a-4eee-b14c-2a18465da204\") " pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:20.019766 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.019737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4d0b72-d67a-4eee-b14c-2a18465da204-tls-certs\") pod \"model-serving-api-54d89bd5fc-crbzt\" (UID: \"da4d0b72-d67a-4eee-b14c-2a18465da204\") " pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:20.025779 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.025757    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rkkq\" (UniqueName: \"kubernetes.io/projected/da4d0b72-d67a-4eee-b14c-2a18465da204-kube-api-access-8rkkq\") pod \"model-serving-api-54d89bd5fc-crbzt\" (UID: \"da4d0b72-d67a-4eee-b14c-2a18465da204\") " pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:20.081963 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.081930    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"]
Jul 24 15:47:20.085562 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.085535    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.088079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.088056    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9pcqq\""
Jul 24 15:47:20.088079 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.088058    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 15:47:20.088296 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.088103    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 24 15:47:20.088296 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.088115    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 24 15:47:20.088296 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.088059    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 24 15:47:20.097387 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.097335    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"]
Jul 24 15:47:20.118484 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.118398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57511a69-5214-4c5f-a052-4b062af85451-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.118484 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.118459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.118717 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.118494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.118717 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.118542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndxp\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-kube-api-access-jndxp\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.118717 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.118573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.118717 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.118604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.118717 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.118673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.191438 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.191403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:20.219688 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.219591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndxp\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-kube-api-access-jndxp\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.219688 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.219642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.219688 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.219663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.219986 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.219712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.219986 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.219737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57511a69-5214-4c5f-a052-4b062af85451-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.219986 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.219771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.219986 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.219805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.220776 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.220726    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.222932 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.222900    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.223043 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.223004    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.223092 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.223037    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.223180 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.223164    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57511a69-5214-4c5f-a052-4b062af85451-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.233769 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.233745    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndxp\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-kube-api-access-jndxp\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.238068 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.238028    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wqgxf\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.345384 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.345349    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-crbzt"]
Jul 24 15:47:20.347932 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:47:20.347902    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4d0b72_d67a_4eee_b14c_2a18465da204.slice/crio-2c28880f376e0f3ed31309ac54a2c7be1657458a326be51d564cfee17cef8865 WatchSource:0}: Error finding container 2c28880f376e0f3ed31309ac54a2c7be1657458a326be51d564cfee17cef8865: Status 404 returned error can't find the container with id 2c28880f376e0f3ed31309ac54a2c7be1657458a326be51d564cfee17cef8865
Jul 24 15:47:20.399065 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.399031    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:20.561536 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.561506    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"]
Jul 24 15:47:20.564584 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:47:20.564556    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57511a69_5214_4c5f_a052_4b062af85451.slice/crio-58eb3947813ee90d57d9e62dd5563c92486aa1aa11cf5f5e6c925d81f74f3e72 WatchSource:0}: Error finding container 58eb3947813ee90d57d9e62dd5563c92486aa1aa11cf5f5e6c925d81f74f3e72: Status 404 returned error can't find the container with id 58eb3947813ee90d57d9e62dd5563c92486aa1aa11cf5f5e6c925d81f74f3e72
Jul 24 15:47:20.748268 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.748168    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-52rtw" event={"ID":"f0abdb77-eff9-4352-b303-9cc347bef506","Type":"ContainerStarted","Data":"fa6b9a2088cf7d88b3325a3e35a8976b52fa1781de952896c519100ab4af1e27"}
Jul 24 15:47:20.750320 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.750221    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf" event={"ID":"57511a69-5214-4c5f-a052-4b062af85451","Type":"ContainerStarted","Data":"58eb3947813ee90d57d9e62dd5563c92486aa1aa11cf5f5e6c925d81f74f3e72"}
Jul 24 15:47:20.752407 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:20.752376    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-crbzt" event={"ID":"da4d0b72-d67a-4eee-b14c-2a18465da204","Type":"ContainerStarted","Data":"2c28880f376e0f3ed31309ac54a2c7be1657458a326be51d564cfee17cef8865"}
Jul 24 15:47:24.975100 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:24.975055    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 15:47:24.975406 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:24.975145    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 15:47:25.786622 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.786582    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf" event={"ID":"57511a69-5214-4c5f-a052-4b062af85451","Type":"ContainerStarted","Data":"3da8a636897d45001c018c9e7a1ded037293ac1ea1ebf6b8fafc8ac1b731891d"}
Jul 24 15:47:25.786808 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.786765    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:25.788785 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.788749    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-crbzt" event={"ID":"da4d0b72-d67a-4eee-b14c-2a18465da204","Type":"ContainerStarted","Data":"76c33d3085be54e03419755cd35b41fae64ae1aaa2ec2775d104657e7bcb42af"}
Jul 24 15:47:25.789286 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.789260    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:25.791131 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.791095    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g" event={"ID":"96fa6894-d9b9-4531-9bd4-a2b3a01c262f","Type":"ContainerStarted","Data":"5dc2ca0feab4ab16e06874b3b13b572514b217e9f0771159b9d087d664675b5a"}
Jul 24 15:47:25.792733 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.792706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-52rtw" event={"ID":"f0abdb77-eff9-4352-b303-9cc347bef506","Type":"ContainerStarted","Data":"0f2da8a60b8c0e5d2a14a68c383be1a8816a2b84482c958b404beba36ac52945"}
Jul 24 15:47:25.792851 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.792838    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:25.811292 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.811226    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf" podStartSLOduration=1.403221397 podStartE2EDuration="5.811208839s" totalImagesPullingTime="4.407987442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:25.809811055 +0000 UTC m=+405.294371655" watchObservedRunningTime="2026-07-24 15:47:25.811208839 +0000 UTC m=+405.295769437"
Jul 24 15:47:25.832233 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.832185    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5688cff95c-qjz2g" podStartSLOduration=2.258271456 podStartE2EDuration="7.832170604s" totalImagesPullingTime="5.573899148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:25.831812896 +0000 UTC m=+405.316373522" watchObservedRunningTime="2026-07-24 15:47:25.832170604 +0000 UTC m=+405.316731271"
Jul 24 15:47:25.855609 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.855562    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-52rtw" podStartSLOduration=2.905593114 podStartE2EDuration="7.855544849s" totalImagesPullingTime="4.949951735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:25.853979199 +0000 UTC m=+405.338539797" watchObservedRunningTime="2026-07-24 15:47:25.855544849 +0000 UTC m=+405.340105446"
Jul 24 15:47:25.872496 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:25.872441    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-crbzt" podStartSLOduration=2.3127461990000002 podStartE2EDuration="6.872420419s" totalImagesPullingTime="4.55967422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:25.870815603 +0000 UTC m=+405.355376204" watchObservedRunningTime="2026-07-24 15:47:25.872420419 +0000 UTC m=+405.356981018"
Jul 24 15:47:26.800087 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:26.800061    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:28.529455 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.529420    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"]
Jul 24 15:47:28.533030 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.533012    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.536359 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.536333    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j5m9b\""
Jul 24 15:47:28.541366 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.541345    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 15:47:28.542676 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.542649    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 15:47:28.542676 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.542670    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 15:47:28.542838 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.542672    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 15:47:28.542965 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.542931    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 15:47:28.548808 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.548785    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"]
Jul 24 15:47:28.607953 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.607918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68141270-4a21-4d80-84e7-5d0be3959fda-cert\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.608126 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.607958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hlp\" (UniqueName: \"kubernetes.io/projected/68141270-4a21-4d80-84e7-5d0be3959fda-kube-api-access-b5hlp\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.608126 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.608013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/68141270-4a21-4d80-84e7-5d0be3959fda-manager-config\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.608126 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.608038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/68141270-4a21-4d80-84e7-5d0be3959fda-metrics-cert\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.708582 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.708545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68141270-4a21-4d80-84e7-5d0be3959fda-cert\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.708582 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.708584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hlp\" (UniqueName: \"kubernetes.io/projected/68141270-4a21-4d80-84e7-5d0be3959fda-kube-api-access-b5hlp\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.708826 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.708724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/68141270-4a21-4d80-84e7-5d0be3959fda-manager-config\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.708826 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.708778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/68141270-4a21-4d80-84e7-5d0be3959fda-metrics-cert\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.709370 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.709285    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/68141270-4a21-4d80-84e7-5d0be3959fda-manager-config\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.711147 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.711127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68141270-4a21-4d80-84e7-5d0be3959fda-cert\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.711270 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.711161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/68141270-4a21-4d80-84e7-5d0be3959fda-metrics-cert\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.730219 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.730187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hlp\" (UniqueName: \"kubernetes.io/projected/68141270-4a21-4d80-84e7-5d0be3959fda-kube-api-access-b5hlp\") pod \"lws-controller-manager-65dbfbfd5-mg884\" (UID: \"68141270-4a21-4d80-84e7-5d0be3959fda\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:28.847897 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:28.847812    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:29.011445 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:29.011418    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"]
Jul 24 15:47:29.014629 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:47:29.014602    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68141270_4a21_4d80_84e7_5d0be3959fda.slice/crio-9544501f6cb2bcae6d8864585bd8c6abffee8cdab5e5523e28110a0f6a50ab6a WatchSource:0}: Error finding container 9544501f6cb2bcae6d8864585bd8c6abffee8cdab5e5523e28110a0f6a50ab6a: Status 404 returned error can't find the container with id 9544501f6cb2bcae6d8864585bd8c6abffee8cdab5e5523e28110a0f6a50ab6a
Jul 24 15:47:29.814958 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:29.814921    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884" event={"ID":"68141270-4a21-4d80-84e7-5d0be3959fda","Type":"ContainerStarted","Data":"9544501f6cb2bcae6d8864585bd8c6abffee8cdab5e5523e28110a0f6a50ab6a"}
Jul 24 15:47:30.667454 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:30.667427    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"]
Jul 24 15:47:30.667618 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:30.667597    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf" podUID="57511a69-5214-4c5f-a052-4b062af85451" containerName="discovery" containerID="cri-o://3da8a636897d45001c018c9e7a1ded037293ac1ea1ebf6b8fafc8ac1b731891d" gracePeriod=30
Jul 24 15:47:30.822406 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:30.822368    2576 generic.go:356] "Generic (PLEG): container finished" podID="57511a69-5214-4c5f-a052-4b062af85451" containerID="3da8a636897d45001c018c9e7a1ded037293ac1ea1ebf6b8fafc8ac1b731891d" exitCode=0
Jul 24 15:47:30.822895 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:30.822436    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf" event={"ID":"57511a69-5214-4c5f-a052-4b062af85451","Type":"ContainerDied","Data":"3da8a636897d45001c018c9e7a1ded037293ac1ea1ebf6b8fafc8ac1b731891d"}
Jul 24 15:47:30.824072 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:30.824039    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884" event={"ID":"68141270-4a21-4d80-84e7-5d0be3959fda","Type":"ContainerStarted","Data":"07643068a81d21ae4363e62bc1c17c384441eee7ff7d5925bcf5da521730a94a"}
Jul 24 15:47:30.824219 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:30.824172    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:47:30.861695 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:30.861639    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884" podStartSLOduration=1.2101957030000001 podStartE2EDuration="2.86162237s" totalImagesPullingTime="1.651426667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:30.860484734 +0000 UTC m=+410.345045333" watchObservedRunningTime="2026-07-24 15:47:30.86162237 +0000 UTC m=+410.346182967"
Jul 24 15:47:30.926477 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:30.926426    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:31.037752 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.037717    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jndxp\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-kube-api-access-jndxp\") pod \"57511a69-5214-4c5f-a052-4b062af85451\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") "
Jul 24 15:47:31.037752 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.037755    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-kubeconfig\") pod \"57511a69-5214-4c5f-a052-4b062af85451\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") "
Jul 24 15:47:31.037971 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.037825    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57511a69-5214-4c5f-a052-4b062af85451-local-certs\") pod \"57511a69-5214-4c5f-a052-4b062af85451\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") "
Jul 24 15:47:31.037971 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.037848    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-dns-cert\") pod \"57511a69-5214-4c5f-a052-4b062af85451\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") "
Jul 24 15:47:31.037971 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.037876    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-istio-token\") pod \"57511a69-5214-4c5f-a052-4b062af85451\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") "
Jul 24 15:47:31.037971 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.037918    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-ca-configmap\") pod \"57511a69-5214-4c5f-a052-4b062af85451\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") "
Jul 24 15:47:31.037971 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.037955    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-cacerts\") pod \"57511a69-5214-4c5f-a052-4b062af85451\" (UID: \"57511a69-5214-4c5f-a052-4b062af85451\") "
Jul 24 15:47:31.040757 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.040717    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57511a69-5214-4c5f-a052-4b062af85451-local-certs" (OuterVolumeSpecName: "local-certs") pod "57511a69-5214-4c5f-a052-4b062af85451" (UID: "57511a69-5214-4c5f-a052-4b062af85451"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:47:31.040907 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.040762    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "57511a69-5214-4c5f-a052-4b062af85451" (UID: "57511a69-5214-4c5f-a052-4b062af85451"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:47:31.040907 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.040770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "57511a69-5214-4c5f-a052-4b062af85451" (UID: "57511a69-5214-4c5f-a052-4b062af85451"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:47:31.041014 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.040903    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "57511a69-5214-4c5f-a052-4b062af85451" (UID: "57511a69-5214-4c5f-a052-4b062af85451"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:47:31.041014 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.040955    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-kube-api-access-jndxp" (OuterVolumeSpecName: "kube-api-access-jndxp") pod "57511a69-5214-4c5f-a052-4b062af85451" (UID: "57511a69-5214-4c5f-a052-4b062af85451"). InnerVolumeSpecName "kube-api-access-jndxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:47:31.045275 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.043317    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-istio-token" (OuterVolumeSpecName: "istio-token") pod "57511a69-5214-4c5f-a052-4b062af85451" (UID: "57511a69-5214-4c5f-a052-4b062af85451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:47:31.046897 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.046876    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-cacerts" (OuterVolumeSpecName: "cacerts") pod "57511a69-5214-4c5f-a052-4b062af85451" (UID: "57511a69-5214-4c5f-a052-4b062af85451"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:47:31.138551 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.138525    2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/57511a69-5214-4c5f-a052-4b062af85451-local-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:47:31.138551 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.138548    2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-dns-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:47:31.138551 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.138557    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-istio-token\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:47:31.138780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.138565    2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/57511a69-5214-4c5f-a052-4b062af85451-istio-csr-ca-configmap\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:47:31.138780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.138573    2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-cacerts\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:47:31.138780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.138582    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndxp\" (UniqueName: \"kubernetes.io/projected/57511a69-5214-4c5f-a052-4b062af85451-kube-api-access-jndxp\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:47:31.138780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.138591    2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/57511a69-5214-4c5f-a052-4b062af85451-istio-kubeconfig\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:47:31.829766 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.829731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf" event={"ID":"57511a69-5214-4c5f-a052-4b062af85451","Type":"ContainerDied","Data":"58eb3947813ee90d57d9e62dd5563c92486aa1aa11cf5f5e6c925d81f74f3e72"}
Jul 24 15:47:31.829766 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.829761    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"
Jul 24 15:47:31.830229 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.829816    2576 scope.go:117] "RemoveContainer" containerID="3da8a636897d45001c018c9e7a1ded037293ac1ea1ebf6b8fafc8ac1b731891d"
Jul 24 15:47:31.856796 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.856767    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"]
Jul 24 15:47:31.869791 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:31.869768    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wqgxf"]
Jul 24 15:47:33.005384 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:33.005351    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57511a69-5214-4c5f-a052-4b062af85451" path="/var/lib/kubelet/pods/57511a69-5214-4c5f-a052-4b062af85451/volumes"
Jul 24 15:47:36.846852 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:36.846817    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-52rtw"
Jul 24 15:47:37.806387 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:37.806357    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-crbzt"
Jul 24 15:47:41.832233 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:47:41.832203    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-mg884"
Jul 24 15:48:35.200641 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.200604    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"]
Jul 24 15:48:35.201054 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.200916    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57511a69-5214-4c5f-a052-4b062af85451" containerName="discovery"
Jul 24 15:48:35.201054 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.200926    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="57511a69-5214-4c5f-a052-4b062af85451" containerName="discovery"
Jul 24 15:48:35.201054 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.200982    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="57511a69-5214-4c5f-a052-4b062af85451" containerName="discovery"
Jul 24 15:48:35.203778 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.203763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.206569 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.206542    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 15:48:35.206704 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.206684    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bsdpk\""
Jul 24 15:48:35.207497 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.207482    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 15:48:35.213727 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.213707    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"]
Jul 24 15:48:35.393046 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.393012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.393216 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.393052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmj9\" (UniqueName: \"kubernetes.io/projected/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-kube-api-access-zlmj9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.393216 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.393074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.493642 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.493550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.493642 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.493591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmj9\" (UniqueName: \"kubernetes.io/projected/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-kube-api-access-zlmj9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.493642 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.493610    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.493933 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.493916    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.494045 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.494025    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.504655 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.504628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmj9\" (UniqueName: \"kubernetes.io/projected/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-kube-api-access-zlmj9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.512506 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.512484    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:35.668887 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.668864    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"]
Jul 24 15:48:35.670918 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:48:35.670889    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf4c4e6_6171_434d_8b2a_d0a7fe36f18a.slice/crio-0282b167da16ac0471669bb30461a614d1a7abae7fe36bae99d9f56268332329 WatchSource:0}: Error finding container 0282b167da16ac0471669bb30461a614d1a7abae7fe36bae99d9f56268332329: Status 404 returned error can't find the container with id 0282b167da16ac0471669bb30461a614d1a7abae7fe36bae99d9f56268332329
Jul 24 15:48:35.891212 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.891178    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"]
Jul 24 15:48:35.894610 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.894594    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:35.908413 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.908385    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"]
Jul 24 15:48:35.997104 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.997073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:35.997346 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.997119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjbm\" (UniqueName: \"kubernetes.io/projected/6e4b692e-fd8f-47c6-a5eb-8734b011753a-kube-api-access-8sjbm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:35.997346 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:35.997266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:36.098429 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.098325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:36.098429 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.098388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjbm\" (UniqueName: \"kubernetes.io/projected/6e4b692e-fd8f-47c6-a5eb-8734b011753a-kube-api-access-8sjbm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:36.098674 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.098465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:36.098741 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.098704    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:36.098805 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.098759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:36.108611 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.108590    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjbm\" (UniqueName: \"kubernetes.io/projected/6e4b692e-fd8f-47c6-a5eb-8734b011753a-kube-api-access-8sjbm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:36.129369 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.129336    2576 generic.go:356] "Generic (PLEG): container finished" podID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerID="0a232c596087c30bcbe8da365590a214151472295f375ab14afba0fadf27dcab" exitCode=0
Jul 24 15:48:36.129488 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.129409    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh" event={"ID":"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a","Type":"ContainerDied","Data":"0a232c596087c30bcbe8da365590a214151472295f375ab14afba0fadf27dcab"}
Jul 24 15:48:36.129488 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.129450    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh" event={"ID":"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a","Type":"ContainerStarted","Data":"0282b167da16ac0471669bb30461a614d1a7abae7fe36bae99d9f56268332329"}
Jul 24 15:48:36.203165 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.203133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:36.338933 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.338909    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"]
Jul 24 15:48:36.340774 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:48:36.340739    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4b692e_fd8f_47c6_a5eb_8734b011753a.slice/crio-8938dc7ba1bf00a886841820d24224c57f929b8b8bc9e0b59c9663b1867f51ee WatchSource:0}: Error finding container 8938dc7ba1bf00a886841820d24224c57f929b8b8bc9e0b59c9663b1867f51ee: Status 404 returned error can't find the container with id 8938dc7ba1bf00a886841820d24224c57f929b8b8bc9e0b59c9663b1867f51ee
Jul 24 15:48:36.600234 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.600203    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"]
Jul 24 15:48:36.603586 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.603571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.613570 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.613546    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"]
Jul 24 15:48:36.704413 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.704375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.704585 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.704430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv55h\" (UniqueName: \"kubernetes.io/projected/27d00aee-53a1-4b28-a551-3277d409dc5e-kube-api-access-qv55h\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.704585 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.704516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.804947 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.804912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.805121 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.804987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.805121 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.805035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv55h\" (UniqueName: \"kubernetes.io/projected/27d00aee-53a1-4b28-a551-3277d409dc5e-kube-api-access-qv55h\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.805300 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.805278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.805372 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.805299    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.814787 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.814751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv55h\" (UniqueName: \"kubernetes.io/projected/27d00aee-53a1-4b28-a551-3277d409dc5e-kube-api-access-qv55h\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:36.912269 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:36.912227    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:37.060358 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.060330    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"]
Jul 24 15:48:37.071158 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:48:37.071132    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d00aee_53a1_4b28_a551_3277d409dc5e.slice/crio-cd939249304664e3fa47ca39975798df56d5c23494a91be720878800926ad2e7 WatchSource:0}: Error finding container cd939249304664e3fa47ca39975798df56d5c23494a91be720878800926ad2e7: Status 404 returned error can't find the container with id cd939249304664e3fa47ca39975798df56d5c23494a91be720878800926ad2e7
Jul 24 15:48:37.135588 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.135556    2576 generic.go:356] "Generic (PLEG): container finished" podID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerID="8a2038071ba7e2db862eebbc8acea7ec56d7a2dbb945cb5da7c1ae8eba27a299" exitCode=0
Jul 24 15:48:37.135721 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.135636    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh" event={"ID":"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a","Type":"ContainerDied","Data":"8a2038071ba7e2db862eebbc8acea7ec56d7a2dbb945cb5da7c1ae8eba27a299"}
Jul 24 15:48:37.137492 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.137462    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc" event={"ID":"27d00aee-53a1-4b28-a551-3277d409dc5e","Type":"ContainerStarted","Data":"d976b3678233051a6a2bcef9030ff796808d62916a18955b688c64a5a3dc2bf4"}
Jul 24 15:48:37.137599 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.137524    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc" event={"ID":"27d00aee-53a1-4b28-a551-3277d409dc5e","Type":"ContainerStarted","Data":"cd939249304664e3fa47ca39975798df56d5c23494a91be720878800926ad2e7"}
Jul 24 15:48:37.139058 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.139039    2576 generic.go:356] "Generic (PLEG): container finished" podID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerID="a71fcd07cbb3a3cb3eff5ba53d588effc5fa1ecd94e6648c9e3b7dced37c5c39" exitCode=0
Jul 24 15:48:37.139145 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.139068    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn" event={"ID":"6e4b692e-fd8f-47c6-a5eb-8734b011753a","Type":"ContainerDied","Data":"a71fcd07cbb3a3cb3eff5ba53d588effc5fa1ecd94e6648c9e3b7dced37c5c39"}
Jul 24 15:48:37.139145 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.139096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn" event={"ID":"6e4b692e-fd8f-47c6-a5eb-8734b011753a","Type":"ContainerStarted","Data":"8938dc7ba1bf00a886841820d24224c57f929b8b8bc9e0b59c9663b1867f51ee"}
Jul 24 15:48:37.213145 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.213109    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"]
Jul 24 15:48:37.216838 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.216818    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.227657 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.227629    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"]
Jul 24 15:48:37.309007 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.308974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.309168 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.309034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnf2h\" (UniqueName: \"kubernetes.io/projected/9d3e023d-c365-48a3-a523-5f34db6c3993-kube-api-access-pnf2h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.309168 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.309081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.409621 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.409526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnf2h\" (UniqueName: \"kubernetes.io/projected/9d3e023d-c365-48a3-a523-5f34db6c3993-kube-api-access-pnf2h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.409621 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.409573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.409621 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.409618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.409979 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.409964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.410030 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.409994    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.420022 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.419994    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnf2h\" (UniqueName: \"kubernetes.io/projected/9d3e023d-c365-48a3-a523-5f34db6c3993-kube-api-access-pnf2h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.537216 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.537174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:37.683929 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:37.683905    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"]
Jul 24 15:48:37.685635 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:48:37.685602    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3e023d_c365_48a3_a523_5f34db6c3993.slice/crio-08a88652b25a2dcdf0e7903060ab873c99c234b6fccd565f3a92e9aaaa87dc4f WatchSource:0}: Error finding container 08a88652b25a2dcdf0e7903060ab873c99c234b6fccd565f3a92e9aaaa87dc4f: Status 404 returned error can't find the container with id 08a88652b25a2dcdf0e7903060ab873c99c234b6fccd565f3a92e9aaaa87dc4f
Jul 24 15:48:38.145442 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:38.145397    2576 generic.go:356] "Generic (PLEG): container finished" podID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerID="8d938490dcaa0dca676d8abe3071c1f3affc78750e5eabfd8c30652c246bfe16" exitCode=0
Jul 24 15:48:38.145634 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:38.145458    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh" event={"ID":"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a","Type":"ContainerDied","Data":"8d938490dcaa0dca676d8abe3071c1f3affc78750e5eabfd8c30652c246bfe16"}
Jul 24 15:48:38.146966 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:38.146937    2576 generic.go:356] "Generic (PLEG): container finished" podID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerID="d976b3678233051a6a2bcef9030ff796808d62916a18955b688c64a5a3dc2bf4" exitCode=0
Jul 24 15:48:38.147073 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:38.146981    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc" event={"ID":"27d00aee-53a1-4b28-a551-3277d409dc5e","Type":"ContainerDied","Data":"d976b3678233051a6a2bcef9030ff796808d62916a18955b688c64a5a3dc2bf4"}
Jul 24 15:48:38.148912 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:38.148892    2576 generic.go:356] "Generic (PLEG): container finished" podID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerID="a31ef6dc6958e4330a04ef30a40386bc97531fbbf9b962a1880bbc3c5df0582d" exitCode=0
Jul 24 15:48:38.149051 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:38.148960    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn" event={"ID":"6e4b692e-fd8f-47c6-a5eb-8734b011753a","Type":"ContainerDied","Data":"a31ef6dc6958e4330a04ef30a40386bc97531fbbf9b962a1880bbc3c5df0582d"}
Jul 24 15:48:38.150648 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:38.150626    2576 generic.go:356] "Generic (PLEG): container finished" podID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerID="5e6fc5df0f51171b794288f92eccab93702693603fa34bb13a74365e7e6488fb" exitCode=0
Jul 24 15:48:38.150729 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:38.150689    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg" event={"ID":"9d3e023d-c365-48a3-a523-5f34db6c3993","Type":"ContainerDied","Data":"5e6fc5df0f51171b794288f92eccab93702693603fa34bb13a74365e7e6488fb"}
Jul 24 15:48:38.150779 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:38.150734    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg" event={"ID":"9d3e023d-c365-48a3-a523-5f34db6c3993","Type":"ContainerStarted","Data":"08a88652b25a2dcdf0e7903060ab873c99c234b6fccd565f3a92e9aaaa87dc4f"}
Jul 24 15:48:39.157410 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.157382    2576 generic.go:356] "Generic (PLEG): container finished" podID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerID="d1b9f0ea20c8b1fb259acdeab848e0ab89fcd47e3af86b027927e53fefa16ee6" exitCode=0
Jul 24 15:48:39.157786 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.157471    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc" event={"ID":"27d00aee-53a1-4b28-a551-3277d409dc5e","Type":"ContainerDied","Data":"d1b9f0ea20c8b1fb259acdeab848e0ab89fcd47e3af86b027927e53fefa16ee6"}
Jul 24 15:48:39.159771 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.159751    2576 generic.go:356] "Generic (PLEG): container finished" podID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerID="3ecdb709494ac0eea2d3ae4fe9491c9831cff9822e3e298606635ac90c749459" exitCode=0
Jul 24 15:48:39.159862 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.159820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn" event={"ID":"6e4b692e-fd8f-47c6-a5eb-8734b011753a","Type":"ContainerDied","Data":"3ecdb709494ac0eea2d3ae4fe9491c9831cff9822e3e298606635ac90c749459"}
Jul 24 15:48:39.161502 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.161478    2576 generic.go:356] "Generic (PLEG): container finished" podID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerID="fd5fc746353f4eb2233a8d9b2f7f45959bc0c068831b525b609ae6aa2d1bdd96" exitCode=0
Jul 24 15:48:39.161600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.161566    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg" event={"ID":"9d3e023d-c365-48a3-a523-5f34db6c3993","Type":"ContainerDied","Data":"fd5fc746353f4eb2233a8d9b2f7f45959bc0c068831b525b609ae6aa2d1bdd96"}
Jul 24 15:48:39.303323 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.303296    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:39.332391 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.332304    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-util\") pod \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") "
Jul 24 15:48:39.334084 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.334060    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-bundle\") pod \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") "
Jul 24 15:48:39.334283 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.334266    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlmj9\" (UniqueName: \"kubernetes.io/projected/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-kube-api-access-zlmj9\") pod \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\" (UID: \"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a\") "
Jul 24 15:48:39.337499 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.337399    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-bundle" (OuterVolumeSpecName: "bundle") pod "4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" (UID: "4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:48:39.339382 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.339350    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-kube-api-access-zlmj9" (OuterVolumeSpecName: "kube-api-access-zlmj9") pod "4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" (UID: "4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a"). InnerVolumeSpecName "kube-api-access-zlmj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:48:39.343418 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.343387    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-util" (OuterVolumeSpecName: "util") pod "4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" (UID: "4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:48:39.435204 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.435174    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-util\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:39.435384 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.435222    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:39.435384 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:39.435233    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlmj9\" (UniqueName: \"kubernetes.io/projected/4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a-kube-api-access-zlmj9\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:40.170358 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.170324    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh" event={"ID":"4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a","Type":"ContainerDied","Data":"0282b167da16ac0471669bb30461a614d1a7abae7fe36bae99d9f56268332329"}
Jul 24 15:48:40.170796 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.170375    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0282b167da16ac0471669bb30461a614d1a7abae7fe36bae99d9f56268332329"
Jul 24 15:48:40.170796 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.170334    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh"
Jul 24 15:48:40.172441 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.172411    2576 generic.go:356] "Generic (PLEG): container finished" podID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerID="82a963d8c8d9e5b8831dca4c960a7b276e6a5704798bdfe3c7fcc36a572410f4" exitCode=0
Jul 24 15:48:40.172544 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.172484    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc" event={"ID":"27d00aee-53a1-4b28-a551-3277d409dc5e","Type":"ContainerDied","Data":"82a963d8c8d9e5b8831dca4c960a7b276e6a5704798bdfe3c7fcc36a572410f4"}
Jul 24 15:48:40.174437 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.174415    2576 generic.go:356] "Generic (PLEG): container finished" podID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerID="6d906e15ddc1523fa65d4c228536ea685c79ea7462585364fcba3f40dbf42cc8" exitCode=0
Jul 24 15:48:40.174554 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.174493    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg" event={"ID":"9d3e023d-c365-48a3-a523-5f34db6c3993","Type":"ContainerDied","Data":"6d906e15ddc1523fa65d4c228536ea685c79ea7462585364fcba3f40dbf42cc8"}
Jul 24 15:48:40.316650 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.316627    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:40.344380 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.344354    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-bundle\") pod \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") "
Jul 24 15:48:40.344544 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.344397    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sjbm\" (UniqueName: \"kubernetes.io/projected/6e4b692e-fd8f-47c6-a5eb-8734b011753a-kube-api-access-8sjbm\") pod \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") "
Jul 24 15:48:40.344544 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.344443    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-util\") pod \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\" (UID: \"6e4b692e-fd8f-47c6-a5eb-8734b011753a\") "
Jul 24 15:48:40.344940 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.344913    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-bundle" (OuterVolumeSpecName: "bundle") pod "6e4b692e-fd8f-47c6-a5eb-8734b011753a" (UID: "6e4b692e-fd8f-47c6-a5eb-8734b011753a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:48:40.346857 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.346831    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4b692e-fd8f-47c6-a5eb-8734b011753a-kube-api-access-8sjbm" (OuterVolumeSpecName: "kube-api-access-8sjbm") pod "6e4b692e-fd8f-47c6-a5eb-8734b011753a" (UID: "6e4b692e-fd8f-47c6-a5eb-8734b011753a"). InnerVolumeSpecName "kube-api-access-8sjbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:48:40.350050 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.350021    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-util" (OuterVolumeSpecName: "util") pod "6e4b692e-fd8f-47c6-a5eb-8734b011753a" (UID: "6e4b692e-fd8f-47c6-a5eb-8734b011753a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:48:40.445168 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.445085    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:40.445168 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.445114    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjbm\" (UniqueName: \"kubernetes.io/projected/6e4b692e-fd8f-47c6-a5eb-8734b011753a-kube-api-access-8sjbm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:40.445168 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:40.445126    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b692e-fd8f-47c6-a5eb-8734b011753a-util\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:41.181317 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.181281    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn" event={"ID":"6e4b692e-fd8f-47c6-a5eb-8734b011753a","Type":"ContainerDied","Data":"8938dc7ba1bf00a886841820d24224c57f929b8b8bc9e0b59c9663b1867f51ee"}
Jul 24 15:48:41.181801 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.181325    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn"
Jul 24 15:48:41.181801 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.181339    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8938dc7ba1bf00a886841820d24224c57f929b8b8bc9e0b59c9663b1867f51ee"
Jul 24 15:48:41.323766 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.323733    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:41.342829 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.342804    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:48:41.352458 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.352433    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv55h\" (UniqueName: \"kubernetes.io/projected/27d00aee-53a1-4b28-a551-3277d409dc5e-kube-api-access-qv55h\") pod \"27d00aee-53a1-4b28-a551-3277d409dc5e\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") "
Jul 24 15:48:41.352579 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.352488    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-bundle\") pod \"27d00aee-53a1-4b28-a551-3277d409dc5e\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") "
Jul 24 15:48:41.352579 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.352519    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-util\") pod \"27d00aee-53a1-4b28-a551-3277d409dc5e\" (UID: \"27d00aee-53a1-4b28-a551-3277d409dc5e\") "
Jul 24 15:48:41.353039 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.353012    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-bundle" (OuterVolumeSpecName: "bundle") pod "27d00aee-53a1-4b28-a551-3277d409dc5e" (UID: "27d00aee-53a1-4b28-a551-3277d409dc5e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:48:41.354684 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.354642    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d00aee-53a1-4b28-a551-3277d409dc5e-kube-api-access-qv55h" (OuterVolumeSpecName: "kube-api-access-qv55h") pod "27d00aee-53a1-4b28-a551-3277d409dc5e" (UID: "27d00aee-53a1-4b28-a551-3277d409dc5e"). InnerVolumeSpecName "kube-api-access-qv55h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:48:41.357444 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.357418    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-util" (OuterVolumeSpecName: "util") pod "27d00aee-53a1-4b28-a551-3277d409dc5e" (UID: "27d00aee-53a1-4b28-a551-3277d409dc5e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:48:41.453887 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.453850    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-util\") pod \"9d3e023d-c365-48a3-a523-5f34db6c3993\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") "
Jul 24 15:48:41.454039 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.453926    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnf2h\" (UniqueName: \"kubernetes.io/projected/9d3e023d-c365-48a3-a523-5f34db6c3993-kube-api-access-pnf2h\") pod \"9d3e023d-c365-48a3-a523-5f34db6c3993\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") "
Jul 24 15:48:41.454039 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.453960    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-bundle\") pod \"9d3e023d-c365-48a3-a523-5f34db6c3993\" (UID: \"9d3e023d-c365-48a3-a523-5f34db6c3993\") "
Jul 24 15:48:41.454153 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.454138    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:41.454190 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.454159    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27d00aee-53a1-4b28-a551-3277d409dc5e-util\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:41.454190 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.454174    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv55h\" (UniqueName: \"kubernetes.io/projected/27d00aee-53a1-4b28-a551-3277d409dc5e-kube-api-access-qv55h\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:41.454615 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.454590    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-bundle" (OuterVolumeSpecName: "bundle") pod "9d3e023d-c365-48a3-a523-5f34db6c3993" (UID: "9d3e023d-c365-48a3-a523-5f34db6c3993"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:48:41.456180 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.456160    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3e023d-c365-48a3-a523-5f34db6c3993-kube-api-access-pnf2h" (OuterVolumeSpecName: "kube-api-access-pnf2h") pod "9d3e023d-c365-48a3-a523-5f34db6c3993" (UID: "9d3e023d-c365-48a3-a523-5f34db6c3993"). InnerVolumeSpecName "kube-api-access-pnf2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:48:41.459357 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.459321    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-util" (OuterVolumeSpecName: "util") pod "9d3e023d-c365-48a3-a523-5f34db6c3993" (UID: "9d3e023d-c365-48a3-a523-5f34db6c3993"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 15:48:41.554775 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.554740    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-util\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:41.554775 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.554771    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnf2h\" (UniqueName: \"kubernetes.io/projected/9d3e023d-c365-48a3-a523-5f34db6c3993-kube-api-access-pnf2h\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:41.554775 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:41.554782    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d3e023d-c365-48a3-a523-5f34db6c3993-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:48:42.188172 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:42.188130    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc" event={"ID":"27d00aee-53a1-4b28-a551-3277d409dc5e","Type":"ContainerDied","Data":"cd939249304664e3fa47ca39975798df56d5c23494a91be720878800926ad2e7"}
Jul 24 15:48:42.188605 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:42.188185    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd939249304664e3fa47ca39975798df56d5c23494a91be720878800926ad2e7"
Jul 24 15:48:42.188605 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:42.188184    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc"
Jul 24 15:48:42.189946 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:42.189923    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg" event={"ID":"9d3e023d-c365-48a3-a523-5f34db6c3993","Type":"ContainerDied","Data":"08a88652b25a2dcdf0e7903060ab873c99c234b6fccd565f3a92e9aaaa87dc4f"}
Jul 24 15:48:42.190044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:42.189961    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08a88652b25a2dcdf0e7903060ab873c99c234b6fccd565f3a92e9aaaa87dc4f"
Jul 24 15:48:42.190044 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:48:42.189972    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg"
Jul 24 15:49:03.353308 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.353271    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd57fc997-lhhch"]
Jul 24 15:49:03.471154 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471123    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"]
Jul 24 15:49:03.471465 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471450    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerName="pull"
Jul 24 15:49:03.471510 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471469    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerName="pull"
Jul 24 15:49:03.471510 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471482    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerName="extract"
Jul 24 15:49:03.471510 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471490    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerName="extract"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471510    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerName="util"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471519    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerName="util"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471528    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerName="util"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471534    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerName="util"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471539    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerName="extract"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471543    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerName="extract"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471551    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerName="util"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471556    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerName="util"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471561    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerName="extract"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471566    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerName="extract"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471574    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerName="util"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471578    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerName="util"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471585    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerName="pull"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471589    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerName="pull"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471596    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerName="pull"
Jul 24 15:49:03.471600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471601    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerName="pull"
Jul 24 15:49:03.472055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471609    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerName="pull"
Jul 24 15:49:03.472055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471615    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerName="pull"
Jul 24 15:49:03.472055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471624    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerName="extract"
Jul 24 15:49:03.472055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471628    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerName="extract"
Jul 24 15:49:03.472055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471690    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d00aee-53a1-4b28-a551-3277d409dc5e" containerName="extract"
Jul 24 15:49:03.472055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471703    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a" containerName="extract"
Jul 24 15:49:03.472055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471714    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4b692e-fd8f-47c6-a5eb-8734b011753a" containerName="extract"
Jul 24 15:49:03.472055 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.471722    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d3e023d-c365-48a3-a523-5f34db6c3993" containerName="extract"
Jul 24 15:49:03.474610 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.474594    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:03.477600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.477578    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bsdpk\""
Jul 24 15:49:03.477752 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.477736    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 24 15:49:03.477806 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.477782    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 15:49:03.477806 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.477800    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 15:49:03.477887 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.477867    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 24 15:49:03.491230 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.491201    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"]
Jul 24 15:49:03.539513 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.539468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ed42646-07f6-400f-92b5-5067c236e340-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:03.539693 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.539600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfx8t\" (UniqueName: \"kubernetes.io/projected/8ed42646-07f6-400f-92b5-5067c236e340-kube-api-access-zfx8t\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:03.539693 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.539681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ed42646-07f6-400f-92b5-5067c236e340-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:03.640638 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.640552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfx8t\" (UniqueName: \"kubernetes.io/projected/8ed42646-07f6-400f-92b5-5067c236e340-kube-api-access-zfx8t\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:03.640638 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.640605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ed42646-07f6-400f-92b5-5067c236e340-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:03.640824 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.640656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ed42646-07f6-400f-92b5-5067c236e340-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:03.640824 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:49:03.640744    2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 24 15:49:03.640824 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:49:03.640820    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed42646-07f6-400f-92b5-5067c236e340-plugin-serving-cert podName:8ed42646-07f6-400f-92b5-5067c236e340 nodeName:}" failed. No retries permitted until 2026-07-24 15:49:04.140801867 +0000 UTC m=+503.625362445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/8ed42646-07f6-400f-92b5-5067c236e340-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-s7rmt" (UID: "8ed42646-07f6-400f-92b5-5067c236e340") : secret "plugin-serving-cert" not found
Jul 24 15:49:03.641411 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.641387    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8ed42646-07f6-400f-92b5-5067c236e340-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:03.659506 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:03.659472    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfx8t\" (UniqueName: \"kubernetes.io/projected/8ed42646-07f6-400f-92b5-5067c236e340-kube-api-access-zfx8t\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:04.145511 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:04.145474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ed42646-07f6-400f-92b5-5067c236e340-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:04.147942 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:04.147924    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ed42646-07f6-400f-92b5-5067c236e340-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-s7rmt\" (UID: \"8ed42646-07f6-400f-92b5-5067c236e340\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:04.383976 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:04.383947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"
Jul 24 15:49:04.531488 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:04.531463    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt"]
Jul 24 15:49:04.533419 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:49:04.533395    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ed42646_07f6_400f_92b5_5067c236e340.slice/crio-eb836e95ea468faa53a385cda170fda5b9c9e4e60ef84fa335f71ef991ad33a0 WatchSource:0}: Error finding container eb836e95ea468faa53a385cda170fda5b9c9e4e60ef84fa335f71ef991ad33a0: Status 404 returned error can't find the container with id eb836e95ea468faa53a385cda170fda5b9c9e4e60ef84fa335f71ef991ad33a0
Jul 24 15:49:05.311807 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:05.311748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt" event={"ID":"8ed42646-07f6-400f-92b5-5067c236e340","Type":"ContainerStarted","Data":"eb836e95ea468faa53a385cda170fda5b9c9e4e60ef84fa335f71ef991ad33a0"}
Jul 24 15:49:26.448366 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:26.448329    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt" event={"ID":"8ed42646-07f6-400f-92b5-5067c236e340","Type":"ContainerStarted","Data":"fa75b04017dfee03680e80aa7ee9c7319fd739747554f932f8b82f2098228b6b"}
Jul 24 15:49:26.474062 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:26.474009    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-s7rmt" podStartSLOduration=2.3045528920000002 podStartE2EDuration="23.47399315s" totalImagesPullingTime="21.169440258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:26.472313851 +0000 UTC m=+525.956874453" watchObservedRunningTime="2026-07-24 15:49:26.47399315 +0000 UTC m=+525.958553788"
Jul 24 15:49:28.372746 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.372700    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fd57fc997-lhhch" podUID="b956f06d-2df7-4010-bf38-44f7c39b2102" containerName="console" containerID="cri-o://1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b" gracePeriod=15
Jul 24 15:49:28.624354 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.624298    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:49:28.662295 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662271    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-oauth-config\") pod \"b956f06d-2df7-4010-bf38-44f7c39b2102\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") "
Jul 24 15:49:28.662427 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662302    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-serving-cert\") pod \"b956f06d-2df7-4010-bf38-44f7c39b2102\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") "
Jul 24 15:49:28.662427 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662324    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ww7s\" (UniqueName: \"kubernetes.io/projected/b956f06d-2df7-4010-bf38-44f7c39b2102-kube-api-access-6ww7s\") pod \"b956f06d-2df7-4010-bf38-44f7c39b2102\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") "
Jul 24 15:49:28.662427 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-service-ca\") pod \"b956f06d-2df7-4010-bf38-44f7c39b2102\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") "
Jul 24 15:49:28.662427 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662367    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-oauth-serving-cert\") pod \"b956f06d-2df7-4010-bf38-44f7c39b2102\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") "
Jul 24 15:49:28.662427 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662397    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-trusted-ca-bundle\") pod \"b956f06d-2df7-4010-bf38-44f7c39b2102\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") "
Jul 24 15:49:28.662679 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662441    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-console-config\") pod \"b956f06d-2df7-4010-bf38-44f7c39b2102\" (UID: \"b956f06d-2df7-4010-bf38-44f7c39b2102\") "
Jul 24 15:49:28.662894 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662849    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-service-ca" (OuterVolumeSpecName: "service-ca") pod "b956f06d-2df7-4010-bf38-44f7c39b2102" (UID: "b956f06d-2df7-4010-bf38-44f7c39b2102"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:49:28.662894 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662872    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b956f06d-2df7-4010-bf38-44f7c39b2102" (UID: "b956f06d-2df7-4010-bf38-44f7c39b2102"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:49:28.662894 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662870    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b956f06d-2df7-4010-bf38-44f7c39b2102" (UID: "b956f06d-2df7-4010-bf38-44f7c39b2102"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:49:28.663031 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.662921    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-console-config" (OuterVolumeSpecName: "console-config") pod "b956f06d-2df7-4010-bf38-44f7c39b2102" (UID: "b956f06d-2df7-4010-bf38-44f7c39b2102"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 15:49:28.664773 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.664746    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b956f06d-2df7-4010-bf38-44f7c39b2102" (UID: "b956f06d-2df7-4010-bf38-44f7c39b2102"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:49:28.665015 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.664994    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b956f06d-2df7-4010-bf38-44f7c39b2102" (UID: "b956f06d-2df7-4010-bf38-44f7c39b2102"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:49:28.665115 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.665016    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b956f06d-2df7-4010-bf38-44f7c39b2102-kube-api-access-6ww7s" (OuterVolumeSpecName: "kube-api-access-6ww7s") pod "b956f06d-2df7-4010-bf38-44f7c39b2102" (UID: "b956f06d-2df7-4010-bf38-44f7c39b2102"). InnerVolumeSpecName "kube-api-access-6ww7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:49:28.763342 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.763307    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ww7s\" (UniqueName: \"kubernetes.io/projected/b956f06d-2df7-4010-bf38-44f7c39b2102-kube-api-access-6ww7s\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:49:28.763342 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.763336    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:49:28.763342 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.763345    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-oauth-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:49:28.763573 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.763354    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-trusted-ca-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:49:28.763573 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.763362    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b956f06d-2df7-4010-bf38-44f7c39b2102-console-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:49:28.763573 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.763370    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-oauth-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:49:28.763573 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:28.763378    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b956f06d-2df7-4010-bf38-44f7c39b2102-console-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:49:29.464024 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:29.463989    2576 generic.go:356] "Generic (PLEG): container finished" podID="b956f06d-2df7-4010-bf38-44f7c39b2102" containerID="1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b" exitCode=0
Jul 24 15:49:29.464456 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:29.464035    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd57fc997-lhhch"
Jul 24 15:49:29.464456 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:29.464078    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd57fc997-lhhch" event={"ID":"b956f06d-2df7-4010-bf38-44f7c39b2102","Type":"ContainerDied","Data":"1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b"}
Jul 24 15:49:29.464456 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:29.464131    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd57fc997-lhhch" event={"ID":"b956f06d-2df7-4010-bf38-44f7c39b2102","Type":"ContainerDied","Data":"847914e2e6aade03171df349b6d3ac7cd411c27c51906b3dfcf17c4b10511587"}
Jul 24 15:49:29.464456 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:29.464158    2576 scope.go:117] "RemoveContainer" containerID="1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b"
Jul 24 15:49:29.472959 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:29.472937    2576 scope.go:117] "RemoveContainer" containerID="1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b"
Jul 24 15:49:29.473193 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:49:29.473171    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b\": container with ID starting with 1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b not found: ID does not exist" containerID="1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b"
Jul 24 15:49:29.473307 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:29.473199    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b"} err="failed to get container status \"1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b\": rpc error: code = NotFound desc = could not find container \"1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b\": container with ID starting with 1c646d0728b752fe605d94aa686517616996b57472bc3f56afa82b5069dfd80b not found: ID does not exist"
Jul 24 15:49:29.483955 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:29.483933    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd57fc997-lhhch"]
Jul 24 15:49:29.490222 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:29.490200    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fd57fc997-lhhch"]
Jul 24 15:49:31.005028 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:31.004981    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b956f06d-2df7-4010-bf38-44f7c39b2102" path="/var/lib/kubelet/pods/b956f06d-2df7-4010-bf38-44f7c39b2102/volumes"
Jul 24 15:49:45.937120 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:45.937086    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:49:45.937523 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:45.937453    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b956f06d-2df7-4010-bf38-44f7c39b2102" containerName="console"
Jul 24 15:49:45.937523 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:45.937464    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b956f06d-2df7-4010-bf38-44f7c39b2102" containerName="console"
Jul 24 15:49:45.937523 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:45.937521    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b956f06d-2df7-4010-bf38-44f7c39b2102" containerName="console"
Jul 24 15:49:46.102508 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.102477    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:49:46.102508 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.102513    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:49:46.102732 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.102584    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:49:46.105524 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.105503    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 15:49:46.186077 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.186045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a04fd46-d9fe-46e0-b2cb-033a886a55c2-config-file\") pod \"limitador-limitador-54d6d988d5-42j5d\" (UID: \"3a04fd46-d9fe-46e0-b2cb-033a886a55c2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:49:46.186234 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.186161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx28m\" (UniqueName: \"kubernetes.io/projected/3a04fd46-d9fe-46e0-b2cb-033a886a55c2-kube-api-access-bx28m\") pod \"limitador-limitador-54d6d988d5-42j5d\" (UID: \"3a04fd46-d9fe-46e0-b2cb-033a886a55c2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:49:46.287504 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.287410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx28m\" (UniqueName: \"kubernetes.io/projected/3a04fd46-d9fe-46e0-b2cb-033a886a55c2-kube-api-access-bx28m\") pod \"limitador-limitador-54d6d988d5-42j5d\" (UID: \"3a04fd46-d9fe-46e0-b2cb-033a886a55c2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:49:46.287504 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.287463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a04fd46-d9fe-46e0-b2cb-033a886a55c2-config-file\") pod \"limitador-limitador-54d6d988d5-42j5d\" (UID: \"3a04fd46-d9fe-46e0-b2cb-033a886a55c2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:49:46.288127 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.288100    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a04fd46-d9fe-46e0-b2cb-033a886a55c2-config-file\") pod \"limitador-limitador-54d6d988d5-42j5d\" (UID: \"3a04fd46-d9fe-46e0-b2cb-033a886a55c2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:49:46.295884 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.295860    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx28m\" (UniqueName: \"kubernetes.io/projected/3a04fd46-d9fe-46e0-b2cb-033a886a55c2-kube-api-access-bx28m\") pod \"limitador-limitador-54d6d988d5-42j5d\" (UID: \"3a04fd46-d9fe-46e0-b2cb-033a886a55c2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:49:46.340179 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.340151    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-l7g4w"]
Jul 24 15:49:46.352018 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.351998    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7g4w"]
Jul 24 15:49:46.352143 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.352087    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l7g4w"
Jul 24 15:49:46.354694 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.354673    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cw7rl\""
Jul 24 15:49:46.388112 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.388075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjgv\" (UniqueName: \"kubernetes.io/projected/a9ee7149-77a0-4c4b-965a-de19dc1a7c8d-kube-api-access-jkjgv\") pod \"authorino-58cd955675-l7g4w\" (UID: \"a9ee7149-77a0-4c4b-965a-de19dc1a7c8d\") " pod="kuadrant-system/authorino-58cd955675-l7g4w"
Jul 24 15:49:46.413175 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.413151    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:49:46.488870 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.488844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjgv\" (UniqueName: \"kubernetes.io/projected/a9ee7149-77a0-4c4b-965a-de19dc1a7c8d-kube-api-access-jkjgv\") pod \"authorino-58cd955675-l7g4w\" (UID: \"a9ee7149-77a0-4c4b-965a-de19dc1a7c8d\") " pod="kuadrant-system/authorino-58cd955675-l7g4w"
Jul 24 15:49:46.497889 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.497863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjgv\" (UniqueName: \"kubernetes.io/projected/a9ee7149-77a0-4c4b-965a-de19dc1a7c8d-kube-api-access-jkjgv\") pod \"authorino-58cd955675-l7g4w\" (UID: \"a9ee7149-77a0-4c4b-965a-de19dc1a7c8d\") " pod="kuadrant-system/authorino-58cd955675-l7g4w"
Jul 24 15:49:46.553994 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.553971    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:49:46.556154 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:49:46.556124    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a04fd46_d9fe_46e0_b2cb_033a886a55c2.slice/crio-1c0193151061694f5ca78e6c931f587dd9f38de2b781b59f7bf64144b72786d6 WatchSource:0}: Error finding container 1c0193151061694f5ca78e6c931f587dd9f38de2b781b59f7bf64144b72786d6: Status 404 returned error can't find the container with id 1c0193151061694f5ca78e6c931f587dd9f38de2b781b59f7bf64144b72786d6
Jul 24 15:49:46.662187 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.662153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l7g4w"
Jul 24 15:49:46.820305 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:46.820200    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7g4w"]
Jul 24 15:49:46.826878 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:49:46.826852    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9ee7149_77a0_4c4b_965a_de19dc1a7c8d.slice/crio-c48365797f627ff076144e2d4374e5924b63c8981662d259506c913193426290 WatchSource:0}: Error finding container c48365797f627ff076144e2d4374e5924b63c8981662d259506c913193426290: Status 404 returned error can't find the container with id c48365797f627ff076144e2d4374e5924b63c8981662d259506c913193426290
Jul 24 15:49:47.553640 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:47.553597    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d" event={"ID":"3a04fd46-d9fe-46e0-b2cb-033a886a55c2","Type":"ContainerStarted","Data":"1c0193151061694f5ca78e6c931f587dd9f38de2b781b59f7bf64144b72786d6"}
Jul 24 15:49:47.555099 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:47.555065    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l7g4w" event={"ID":"a9ee7149-77a0-4c4b-965a-de19dc1a7c8d","Type":"ContainerStarted","Data":"c48365797f627ff076144e2d4374e5924b63c8981662d259506c913193426290"}
Jul 24 15:49:50.202472 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:50.202442    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7g4w"]
Jul 24 15:49:52.595077 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:52.595044    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d" event={"ID":"3a04fd46-d9fe-46e0-b2cb-033a886a55c2","Type":"ContainerStarted","Data":"750ff3a82e5f2a98b0c78c3c2d40a5442c82dbd8f254848e2825b19b2ef3f2f8"}
Jul 24 15:49:52.595481 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:52.595157    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:49:52.636789 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:52.636733    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d" podStartSLOduration=1.7936090089999999 podStartE2EDuration="7.636715089s" totalImagesPullingTime="5.84310608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:52.63527874 +0000 UTC m=+552.119839337" watchObservedRunningTime="2026-07-24 15:49:52.636715089 +0000 UTC m=+552.121275689"
Jul 24 15:49:53.601212 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:53.601173    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l7g4w" event={"ID":"a9ee7149-77a0-4c4b-965a-de19dc1a7c8d","Type":"ContainerStarted","Data":"84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd"}
Jul 24 15:49:53.601687 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:53.601221    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-l7g4w" podUID="a9ee7149-77a0-4c4b-965a-de19dc1a7c8d" containerName="authorino" containerID="cri-o://84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd" gracePeriod=30
Jul 24 15:49:53.617136 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:53.617094    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-l7g4w" podStartSLOduration=1.87295818 podStartE2EDuration="7.617080314s" totalImagesPullingTime="5.744122134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:53.616138787 +0000 UTC m=+553.100699386" watchObservedRunningTime="2026-07-24 15:49:53.617080314 +0000 UTC m=+553.101640911"
Jul 24 15:49:53.900333 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:53.900304    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l7g4w"
Jul 24 15:49:53.956300 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:53.956266    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkjgv\" (UniqueName: \"kubernetes.io/projected/a9ee7149-77a0-4c4b-965a-de19dc1a7c8d-kube-api-access-jkjgv\") pod \"a9ee7149-77a0-4c4b-965a-de19dc1a7c8d\" (UID: \"a9ee7149-77a0-4c4b-965a-de19dc1a7c8d\") "
Jul 24 15:49:53.958544 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:53.958521    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ee7149-77a0-4c4b-965a-de19dc1a7c8d-kube-api-access-jkjgv" (OuterVolumeSpecName: "kube-api-access-jkjgv") pod "a9ee7149-77a0-4c4b-965a-de19dc1a7c8d" (UID: "a9ee7149-77a0-4c4b-965a-de19dc1a7c8d"). InnerVolumeSpecName "kube-api-access-jkjgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:49:54.057092 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.057057    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkjgv\" (UniqueName: \"kubernetes.io/projected/a9ee7149-77a0-4c4b-965a-de19dc1a7c8d-kube-api-access-jkjgv\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:49:54.606978 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.606946    2576 generic.go:356] "Generic (PLEG): container finished" podID="a9ee7149-77a0-4c4b-965a-de19dc1a7c8d" containerID="84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd" exitCode=0
Jul 24 15:49:54.607457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.607001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l7g4w" event={"ID":"a9ee7149-77a0-4c4b-965a-de19dc1a7c8d","Type":"ContainerDied","Data":"84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd"}
Jul 24 15:49:54.607457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.607027    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l7g4w"
Jul 24 15:49:54.607457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.607042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l7g4w" event={"ID":"a9ee7149-77a0-4c4b-965a-de19dc1a7c8d","Type":"ContainerDied","Data":"c48365797f627ff076144e2d4374e5924b63c8981662d259506c913193426290"}
Jul 24 15:49:54.607457 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.607065    2576 scope.go:117] "RemoveContainer" containerID="84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd"
Jul 24 15:49:54.616205 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.616180    2576 scope.go:117] "RemoveContainer" containerID="84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd"
Jul 24 15:49:54.616493 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:49:54.616473    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd\": container with ID starting with 84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd not found: ID does not exist" containerID="84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd"
Jul 24 15:49:54.616548 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.616502    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd"} err="failed to get container status \"84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd\": rpc error: code = NotFound desc = could not find container \"84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd\": container with ID starting with 84297b077fdc29f43ac221daca0583706261025b1c99c1c284e8fd6d1a2bcadd not found: ID does not exist"
Jul 24 15:49:54.627882 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.627853    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7g4w"]
Jul 24 15:49:54.630105 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:54.630085    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-l7g4w"]
Jul 24 15:49:55.004881 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:49:55.004851    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ee7149-77a0-4c4b-965a-de19dc1a7c8d" path="/var/lib/kubelet/pods/a9ee7149-77a0-4c4b-965a-de19dc1a7c8d/volumes"
Jul 24 15:50:03.604060 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:03.604025    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-42j5d"
Jul 24 15:50:20.224904 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.224872    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7k9nc"]
Jul 24 15:50:20.225342 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.225222    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ee7149-77a0-4c4b-965a-de19dc1a7c8d" containerName="authorino"
Jul 24 15:50:20.225342 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.225233    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ee7149-77a0-4c4b-965a-de19dc1a7c8d" containerName="authorino"
Jul 24 15:50:20.225342 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.225307    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ee7149-77a0-4c4b-965a-de19dc1a7c8d" containerName="authorino"
Jul 24 15:50:20.228330 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.228313    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7k9nc"
Jul 24 15:50:20.230909 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.230889    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cw7rl\""
Jul 24 15:50:20.237862 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.237839    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7k9nc"]
Jul 24 15:50:20.380616 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.380579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xm8\" (UniqueName: \"kubernetes.io/projected/2955b876-c7a6-4042-b2cd-9a6feedd4512-kube-api-access-c8xm8\") pod \"authorino-5d4b67c446-7k9nc\" (UID: \"2955b876-c7a6-4042-b2cd-9a6feedd4512\") " pod="kuadrant-system/authorino-5d4b67c446-7k9nc"
Jul 24 15:50:20.465715 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.465681    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7k9nc"]
Jul 24 15:50:20.465944 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:50:20.465925    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-c8xm8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-7k9nc" podUID="2955b876-c7a6-4042-b2cd-9a6feedd4512"
Jul 24 15:50:20.481798 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.481728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xm8\" (UniqueName: \"kubernetes.io/projected/2955b876-c7a6-4042-b2cd-9a6feedd4512-kube-api-access-c8xm8\") pod \"authorino-5d4b67c446-7k9nc\" (UID: \"2955b876-c7a6-4042-b2cd-9a6feedd4512\") " pod="kuadrant-system/authorino-5d4b67c446-7k9nc"
Jul 24 15:50:20.495235 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.495199    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xm8\" (UniqueName: \"kubernetes.io/projected/2955b876-c7a6-4042-b2cd-9a6feedd4512-kube-api-access-c8xm8\") pod \"authorino-5d4b67c446-7k9nc\" (UID: \"2955b876-c7a6-4042-b2cd-9a6feedd4512\") " pod="kuadrant-system/authorino-5d4b67c446-7k9nc"
Jul 24 15:50:20.698319 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.698286    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65ff56fb57-v4qs6"]
Jul 24 15:50:20.708314 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.708288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:50:20.709671 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.709648    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65ff56fb57-v4qs6"]
Jul 24 15:50:20.711324 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.711299    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 15:50:20.737971 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.737895    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7k9nc"
Jul 24 15:50:20.745661 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.745625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7k9nc"
Jul 24 15:50:20.785444 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.785407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jtr\" (UniqueName: \"kubernetes.io/projected/63541c54-8163-45e5-a04a-57ae992a019a-kube-api-access-s6jtr\") pod \"authorino-65ff56fb57-v4qs6\" (UID: \"63541c54-8163-45e5-a04a-57ae992a019a\") " pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:50:20.785444 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.785446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63541c54-8163-45e5-a04a-57ae992a019a-tls-cert\") pod \"authorino-65ff56fb57-v4qs6\" (UID: \"63541c54-8163-45e5-a04a-57ae992a019a\") " pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:50:20.886717 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.886682    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8xm8\" (UniqueName: \"kubernetes.io/projected/2955b876-c7a6-4042-b2cd-9a6feedd4512-kube-api-access-c8xm8\") pod \"2955b876-c7a6-4042-b2cd-9a6feedd4512\" (UID: \"2955b876-c7a6-4042-b2cd-9a6feedd4512\") "
Jul 24 15:50:20.886898 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.886858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jtr\" (UniqueName: \"kubernetes.io/projected/63541c54-8163-45e5-a04a-57ae992a019a-kube-api-access-s6jtr\") pod \"authorino-65ff56fb57-v4qs6\" (UID: \"63541c54-8163-45e5-a04a-57ae992a019a\") " pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:50:20.886898 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.886885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63541c54-8163-45e5-a04a-57ae992a019a-tls-cert\") pod \"authorino-65ff56fb57-v4qs6\" (UID: \"63541c54-8163-45e5-a04a-57ae992a019a\") " pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:50:20.888975 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.888943    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2955b876-c7a6-4042-b2cd-9a6feedd4512-kube-api-access-c8xm8" (OuterVolumeSpecName: "kube-api-access-c8xm8") pod "2955b876-c7a6-4042-b2cd-9a6feedd4512" (UID: "2955b876-c7a6-4042-b2cd-9a6feedd4512"). InnerVolumeSpecName "kube-api-access-c8xm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:50:20.889419 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.889399    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63541c54-8163-45e5-a04a-57ae992a019a-tls-cert\") pod \"authorino-65ff56fb57-v4qs6\" (UID: \"63541c54-8163-45e5-a04a-57ae992a019a\") " pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:50:20.894801 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.894772    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jtr\" (UniqueName: \"kubernetes.io/projected/63541c54-8163-45e5-a04a-57ae992a019a-kube-api-access-s6jtr\") pod \"authorino-65ff56fb57-v4qs6\" (UID: \"63541c54-8163-45e5-a04a-57ae992a019a\") " pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:50:20.988192 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:20.988109    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8xm8\" (UniqueName: \"kubernetes.io/projected/2955b876-c7a6-4042-b2cd-9a6feedd4512-kube-api-access-c8xm8\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:50:21.018890 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:21.018856    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:50:21.178272 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:21.178228    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65ff56fb57-v4qs6"]
Jul 24 15:50:21.180080 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:50:21.180048    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63541c54_8163_45e5_a04a_57ae992a019a.slice/crio-3ec02f132f70023919f1bdf831fa8442beedb46a5fc073e5b06ced9f25b1f673 WatchSource:0}: Error finding container 3ec02f132f70023919f1bdf831fa8442beedb46a5fc073e5b06ced9f25b1f673: Status 404 returned error can't find the container with id 3ec02f132f70023919f1bdf831fa8442beedb46a5fc073e5b06ced9f25b1f673
Jul 24 15:50:21.743914 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:21.743880    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65ff56fb57-v4qs6" event={"ID":"63541c54-8163-45e5-a04a-57ae992a019a","Type":"ContainerStarted","Data":"3ec02f132f70023919f1bdf831fa8442beedb46a5fc073e5b06ced9f25b1f673"}
Jul 24 15:50:21.744214 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:21.743938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7k9nc"
Jul 24 15:50:21.772432 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:21.772399    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7k9nc"]
Jul 24 15:50:21.777017 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:21.776997    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7k9nc"]
Jul 24 15:50:22.750714 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:22.750677    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65ff56fb57-v4qs6" event={"ID":"63541c54-8163-45e5-a04a-57ae992a019a","Type":"ContainerStarted","Data":"1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7"}
Jul 24 15:50:22.767391 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:22.767345    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65ff56fb57-v4qs6" podStartSLOduration=2.281717321 podStartE2EDuration="2.767331598s" totalImagesPullingTime="485.614277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:50:22.766119154 +0000 UTC m=+582.250679752" watchObservedRunningTime="2026-07-24 15:50:22.767331598 +0000 UTC m=+582.251892237"
Jul 24 15:50:23.010785 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:23.010705    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2955b876-c7a6-4042-b2cd-9a6feedd4512" path="/var/lib/kubelet/pods/2955b876-c7a6-4042-b2cd-9a6feedd4512/volumes"
Jul 24 15:50:40.978929 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:40.978898    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 15:50:40.981839 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:40.981805    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 15:50:40.983284 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:40.983263    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:50:40.986448 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:50:40.986431    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:51:04.114259 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:51:04.114203    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:51:33.466817 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:51:33.466784    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:51:54.665640 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:51:54.665607    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:52:21.674592 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:21.674558    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:52:34.661189 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:34.661086    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:52:45.678793 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:45.678755    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:52:53.962397 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:53.962361    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"]
Jul 24 15:52:53.965997 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:53.965980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:53.970143 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:53.970116    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 15:52:53.970295 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:53.970175    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 15:52:53.970295 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:53.970257    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 24 15:52:53.970295 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:53.970265    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lmqq6\""
Jul 24 15:52:53.974583 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:53.974558    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"]
Jul 24 15:52:54.088530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.088487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.088530 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.088534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53206d9b-ec6b-474d-9301-84a0e354fda3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.088780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.088609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.088780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.088633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk526\" (UniqueName: \"kubernetes.io/projected/53206d9b-ec6b-474d-9301-84a0e354fda3-kube-api-access-nk526\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.088780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.088680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.088780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.088702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.189549 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.189512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.189731 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.189558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53206d9b-ec6b-474d-9301-84a0e354fda3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.189731 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.189609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.189731 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.189633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk526\" (UniqueName: \"kubernetes.io/projected/53206d9b-ec6b-474d-9301-84a0e354fda3-kube-api-access-nk526\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.189731 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.189681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.189731 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.189704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.190060 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.190037    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.190060 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.190050    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.190170 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.190084    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.192600 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.192572    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53206d9b-ec6b-474d-9301-84a0e354fda3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.192818 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.192764    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53206d9b-ec6b-474d-9301-84a0e354fda3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.199122 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.199100    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk526\" (UniqueName: \"kubernetes.io/projected/53206d9b-ec6b-474d-9301-84a0e354fda3-kube-api-access-nk526\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4\" (UID: \"53206d9b-ec6b-474d-9301-84a0e354fda3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.276934 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.276857    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:52:54.412898 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.412870    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"]
Jul 24 15:52:54.414754 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:52:54.414717    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53206d9b_ec6b_474d_9301_84a0e354fda3.slice/crio-882a0f8ac9a5c3f44e5747962f9e5d56b269659f69f91188136802d312feca21 WatchSource:0}: Error finding container 882a0f8ac9a5c3f44e5747962f9e5d56b269659f69f91188136802d312feca21: Status 404 returned error can't find the container with id 882a0f8ac9a5c3f44e5747962f9e5d56b269659f69f91188136802d312feca21
Jul 24 15:52:54.416227 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.416203    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 15:52:54.516110 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:54.516073    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4" event={"ID":"53206d9b-ec6b-474d-9301-84a0e354fda3","Type":"ContainerStarted","Data":"882a0f8ac9a5c3f44e5747962f9e5d56b269659f69f91188136802d312feca21"}
Jul 24 15:52:55.523295 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:55.523258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4" event={"ID":"53206d9b-ec6b-474d-9301-84a0e354fda3","Type":"ContainerStarted","Data":"4d34fd3f4c0d0467838a8a3073cd094e783dd1e305216d6349d5d039f487f76b"}
Jul 24 15:52:55.542303 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:55.542256    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4" podStartSLOduration=1.545457001 podStartE2EDuration="2.542227533s" totalImagesPullingTime="996.770532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:55.539460566 +0000 UTC m=+735.024021164" watchObservedRunningTime="2026-07-24 15:52:55.542227533 +0000 UTC m=+735.026788131"
Jul 24 15:52:56.873161 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:52:56.873124    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:53:04.277206 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:53:04.277166    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:53:04.277642 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:53:04.277219    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:53:04.289608 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:53:04.289585    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:53:04.578730 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:53:04.578649    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4"
Jul 24 15:54:17.130122 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.130086    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d58695fd4-kjl7p"]
Jul 24 15:54:17.133301 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.133275    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d58695fd4-kjl7p"
Jul 24 15:54:17.143183 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.143160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d58695fd4-kjl7p"]
Jul 24 15:54:17.200900 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.200869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd9rr\" (UniqueName: \"kubernetes.io/projected/aec2c1d1-6069-4021-a393-e94a9b116163-kube-api-access-fd9rr\") pod \"authorino-7d58695fd4-kjl7p\" (UID: \"aec2c1d1-6069-4021-a393-e94a9b116163\") " pod="kuadrant-system/authorino-7d58695fd4-kjl7p"
Jul 24 15:54:17.201074 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.200914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aec2c1d1-6069-4021-a393-e94a9b116163-tls-cert\") pod \"authorino-7d58695fd4-kjl7p\" (UID: \"aec2c1d1-6069-4021-a393-e94a9b116163\") " pod="kuadrant-system/authorino-7d58695fd4-kjl7p"
Jul 24 15:54:17.301398 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.301361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aec2c1d1-6069-4021-a393-e94a9b116163-tls-cert\") pod \"authorino-7d58695fd4-kjl7p\" (UID: \"aec2c1d1-6069-4021-a393-e94a9b116163\") " pod="kuadrant-system/authorino-7d58695fd4-kjl7p"
Jul 24 15:54:17.301565 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.301443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd9rr\" (UniqueName: \"kubernetes.io/projected/aec2c1d1-6069-4021-a393-e94a9b116163-kube-api-access-fd9rr\") pod \"authorino-7d58695fd4-kjl7p\" (UID: \"aec2c1d1-6069-4021-a393-e94a9b116163\") " pod="kuadrant-system/authorino-7d58695fd4-kjl7p"
Jul 24 15:54:17.304480 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.304450    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aec2c1d1-6069-4021-a393-e94a9b116163-tls-cert\") pod \"authorino-7d58695fd4-kjl7p\" (UID: \"aec2c1d1-6069-4021-a393-e94a9b116163\") " pod="kuadrant-system/authorino-7d58695fd4-kjl7p"
Jul 24 15:54:17.310841 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.310809    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd9rr\" (UniqueName: \"kubernetes.io/projected/aec2c1d1-6069-4021-a393-e94a9b116163-kube-api-access-fd9rr\") pod \"authorino-7d58695fd4-kjl7p\" (UID: \"aec2c1d1-6069-4021-a393-e94a9b116163\") " pod="kuadrant-system/authorino-7d58695fd4-kjl7p"
Jul 24 15:54:17.442328 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.442294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d58695fd4-kjl7p"
Jul 24 15:54:17.586967 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.586941    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d58695fd4-kjl7p"]
Jul 24 15:54:17.589562 ip-10-0-131-210 kubenswrapper[2576]: W0724 15:54:17.589533    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaec2c1d1_6069_4021_a393_e94a9b116163.slice/crio-c09677b26abeb668ce62ca225d6a2b996ad4d05a9a442ca9aee64c2851c5ee73 WatchSource:0}: Error finding container c09677b26abeb668ce62ca225d6a2b996ad4d05a9a442ca9aee64c2851c5ee73: Status 404 returned error can't find the container with id c09677b26abeb668ce62ca225d6a2b996ad4d05a9a442ca9aee64c2851c5ee73
Jul 24 15:54:17.933686 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:17.933657    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d58695fd4-kjl7p" event={"ID":"aec2c1d1-6069-4021-a393-e94a9b116163","Type":"ContainerStarted","Data":"c09677b26abeb668ce62ca225d6a2b996ad4d05a9a442ca9aee64c2851c5ee73"}
Jul 24 15:54:18.941834 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:18.941795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d58695fd4-kjl7p" event={"ID":"aec2c1d1-6069-4021-a393-e94a9b116163","Type":"ContainerStarted","Data":"552e8741b951c19e669370b8e756a4819cbaf48fd2347bc416b570b1b8af473b"}
Jul 24 15:54:18.957969 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:18.957918    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d58695fd4-kjl7p" podStartSLOduration=1.581985492 podStartE2EDuration="1.957898787s" totalImagesPullingTime="375.913295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:18.957643102 +0000 UTC m=+818.442203701" watchObservedRunningTime="2026-07-24 15:54:18.957898787 +0000 UTC m=+818.442459385"
Jul 24 15:54:18.996310 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:18.996274    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65ff56fb57-v4qs6"]
Jul 24 15:54:18.996547 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:18.996495    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-65ff56fb57-v4qs6" podUID="63541c54-8163-45e5-a04a-57ae992a019a" containerName="authorino" containerID="cri-o://1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7" gracePeriod=30
Jul 24 15:54:19.262459 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.262412    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:54:19.318054 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.318017    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6jtr\" (UniqueName: \"kubernetes.io/projected/63541c54-8163-45e5-a04a-57ae992a019a-kube-api-access-s6jtr\") pod \"63541c54-8163-45e5-a04a-57ae992a019a\" (UID: \"63541c54-8163-45e5-a04a-57ae992a019a\") "
Jul 24 15:54:19.318264 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.318064    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63541c54-8163-45e5-a04a-57ae992a019a-tls-cert\") pod \"63541c54-8163-45e5-a04a-57ae992a019a\" (UID: \"63541c54-8163-45e5-a04a-57ae992a019a\") "
Jul 24 15:54:19.320606 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.320574    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63541c54-8163-45e5-a04a-57ae992a019a-kube-api-access-s6jtr" (OuterVolumeSpecName: "kube-api-access-s6jtr") pod "63541c54-8163-45e5-a04a-57ae992a019a" (UID: "63541c54-8163-45e5-a04a-57ae992a019a"). InnerVolumeSpecName "kube-api-access-s6jtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 15:54:19.329645 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.329619    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63541c54-8163-45e5-a04a-57ae992a019a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "63541c54-8163-45e5-a04a-57ae992a019a" (UID: "63541c54-8163-45e5-a04a-57ae992a019a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 15:54:19.418725 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.418691    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6jtr\" (UniqueName: \"kubernetes.io/projected/63541c54-8163-45e5-a04a-57ae992a019a-kube-api-access-s6jtr\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:54:19.418725 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.418720    2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63541c54-8163-45e5-a04a-57ae992a019a-tls-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 15:54:19.949007 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.948974    2576 generic.go:356] "Generic (PLEG): container finished" podID="63541c54-8163-45e5-a04a-57ae992a019a" containerID="1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7" exitCode=0
Jul 24 15:54:19.949467 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.949021    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65ff56fb57-v4qs6"
Jul 24 15:54:19.949467 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.949061    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65ff56fb57-v4qs6" event={"ID":"63541c54-8163-45e5-a04a-57ae992a019a","Type":"ContainerDied","Data":"1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7"}
Jul 24 15:54:19.949467 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.949105    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65ff56fb57-v4qs6" event={"ID":"63541c54-8163-45e5-a04a-57ae992a019a","Type":"ContainerDied","Data":"3ec02f132f70023919f1bdf831fa8442beedb46a5fc073e5b06ced9f25b1f673"}
Jul 24 15:54:19.949467 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.949122    2576 scope.go:117] "RemoveContainer" containerID="1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7"
Jul 24 15:54:19.958963 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.958944    2576 scope.go:117] "RemoveContainer" containerID="1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7"
Jul 24 15:54:19.959305 ip-10-0-131-210 kubenswrapper[2576]: E0724 15:54:19.959269    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7\": container with ID starting with 1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7 not found: ID does not exist" containerID="1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7"
Jul 24 15:54:19.959355 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.959301    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7"} err="failed to get container status \"1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7\": rpc error: code = NotFound desc = could not find container \"1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7\": container with ID starting with 1718bf843d2bb261bb8bed5e1809dfafe1bd048836acbda56b97854041b765a7 not found: ID does not exist"
Jul 24 15:54:19.970108 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.970075    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65ff56fb57-v4qs6"]
Jul 24 15:54:19.972132 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:19.972113    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65ff56fb57-v4qs6"]
Jul 24 15:54:21.005253 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:21.005204    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63541c54-8163-45e5-a04a-57ae992a019a" path="/var/lib/kubelet/pods/63541c54-8163-45e5-a04a-57ae992a019a/volumes"
Jul 24 15:54:48.168750 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:48.168715    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:54:59.673123 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:54:59.673084    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:55:09.570130 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:55:09.570092    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:55:41.023555 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:55:41.023460    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 15:55:41.026928 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:55:41.026904    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 15:55:41.028392 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:55:41.028370    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:55:41.031432 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:55:41.031412    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 15:56:51.158780 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:56:51.158739    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:56:59.258164 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:56:59.258125    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:57:31.664750 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:57:31.664714    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:57:41.063764 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:57:41.063734    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:57:51.671450 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:57:51.671413    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:57:57.365443 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:57:57.365403    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:58:08.366579 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:58:08.366548    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:58:20.866201 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:58:20.866164    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:58:31.469207 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:58:31.469175    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:58:41.469711 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:58:41.469629    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:59:03.065850 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:59:03.065819    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:59:14.762518 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:59:14.762482    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:59:36.673069 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:59:36.673028    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 15:59:47.560696 ip-10-0-131-210 kubenswrapper[2576]: I0724 15:59:47.560648    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:00:04.277829 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:00:04.277780    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:00:13.674048 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:00:13.674013    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:00:23.069771 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:00:23.069736    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:00:32.769381 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:00:32.769335    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:00:41.056471 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:00:41.056442    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:00:41.060921 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:00:41.060895    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:00:41.061805 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:00:41.061784    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:00:41.065562 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:00:41.065544    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:00:42.068358 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:00:42.068328    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:02:30.967822 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:02:30.967786    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:02:41.692439 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:02:41.692405    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:02:50.864309 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:02:50.864268    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:03:02.888920 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:03:02.888826    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:03:14.463344 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:03:14.463314    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:03:23.863204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:03:23.863164    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:03:33.809326 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:03:33.809293    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:03:43.780229 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:03:43.780186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:03:55.876835 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:03:55.876805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:04:04.709795 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:04.709760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:04:15.485877 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:15.485846    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:04:39.664148 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:39.664108    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:04:45.878104 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:45.878071    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"]
Jul 24 16:04:45.878501 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:45.878483    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63541c54-8163-45e5-a04a-57ae992a019a" containerName="authorino"
Jul 24 16:04:45.878501 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:45.878496    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63541c54-8163-45e5-a04a-57ae992a019a" containerName="authorino"
Jul 24 16:04:45.878601 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:45.878590    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63541c54-8163-45e5-a04a-57ae992a019a" containerName="authorino"
Jul 24 16:04:45.881445 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:45.881429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:45.884404 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:45.884384    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5cd8d52e-kserve-self-signed-certs\""
Jul 24 16:04:45.903874 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:45.903849    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"]
Jul 24 16:04:46.035906 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.035874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jss7z\" (UniqueName: \"kubernetes.io/projected/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-kube-api-access-jss7z\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.036066 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.035916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tmp-dir\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.036066 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.035946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-dshm\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.036066 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.036007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-model-cache\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.036066 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.036030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tls-certs\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.036066 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.036063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-home\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.136510 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.136426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-model-cache\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.136510 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.136460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tls-certs\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.136510 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.136490    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-home\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.136815 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.136579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jss7z\" (UniqueName: \"kubernetes.io/projected/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-kube-api-access-jss7z\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.136815 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.136653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tmp-dir\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.136815 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.136695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-dshm\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.136974 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.136859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-model-cache\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.136974 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.136918    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-home\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.137211 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.137183    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tmp-dir\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.139026 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.139007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-dshm\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.139329 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.139310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tls-certs\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.149196 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.149173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jss7z\" (UniqueName: \"kubernetes.io/projected/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-kube-api-access-jss7z\") pod \"e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") " pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.189988 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.189962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:46.337057 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:46.337035    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"]
Jul 24 16:04:46.339706 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:04:46.339684    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8dc82bc_6dc9_42bc_9bcc_1bd26a43b8f6.slice/crio-89a20b0b6542803a3ab4a971f6b1db334646cb5a15e37132f48a5d03ede95aaa WatchSource:0}: Error finding container 89a20b0b6542803a3ab4a971f6b1db334646cb5a15e37132f48a5d03ede95aaa: Status 404 returned error can't find the container with id 89a20b0b6542803a3ab4a971f6b1db334646cb5a15e37132f48a5d03ede95aaa
Jul 24 16:04:47.123442 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:47.123403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45" event={"ID":"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6","Type":"ContainerStarted","Data":"2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95"}
Jul 24 16:04:47.123442 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:47.123457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45" event={"ID":"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6","Type":"ContainerStarted","Data":"89a20b0b6542803a3ab4a971f6b1db334646cb5a15e37132f48a5d03ede95aaa"}
Jul 24 16:04:47.140946 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:47.140904    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45" podStartSLOduration=2.140889006 podStartE2EDuration="2.140889006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:04:47.139784211 +0000 UTC m=+1446.624344803" watchObservedRunningTime="2026-07-24 16:04:47.140889006 +0000 UTC m=+1446.625449604"
Jul 24 16:04:48.165128 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:48.165093    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:04:51.856006 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.855973    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"]
Jul 24 16:04:51.860150 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.860131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:51.862754 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.862733    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5cd8d52e-kserve-self-signed-certs\""
Jul 24 16:04:51.869554 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.869532    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"]
Jul 24 16:04:51.993676 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.993614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82372b79-782b-48d0-b5cc-000500cc6b8b-tls-certs\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:51.993855 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.993683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-tmp-dir\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:51.993855 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.993715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-dshm\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:51.993855 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.993752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-model-cache\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:51.993855 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.993840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgnl\" (UniqueName: \"kubernetes.io/projected/82372b79-782b-48d0-b5cc-000500cc6b8b-kube-api-access-nkgnl\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:51.994011 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:51.993896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-home\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.094408 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.094372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82372b79-782b-48d0-b5cc-000500cc6b8b-tls-certs\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.094585 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.094534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-tmp-dir\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.094638 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.094597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-dshm\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.094698 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.094683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-model-cache\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.094749 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.094733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgnl\" (UniqueName: \"kubernetes.io/projected/82372b79-782b-48d0-b5cc-000500cc6b8b-kube-api-access-nkgnl\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.094840 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.094820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-home\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.095074 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.094969    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-tmp-dir\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.095155 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.095135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-model-cache\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.095200 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.095150    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-home\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.096964 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.096947    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-dshm\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.097025 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.097007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82372b79-782b-48d0-b5cc-000500cc6b8b-tls-certs\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.104528 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.104508    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgnl\" (UniqueName: \"kubernetes.io/projected/82372b79-782b-48d0-b5cc-000500cc6b8b-kube-api-access-nkgnl\") pod \"e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") " pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.170778 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.170746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:04:52.309053 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:52.309024    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"]
Jul 24 16:04:52.311004 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:04:52.310976    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82372b79_782b_48d0_b5cc_000500cc6b8b.slice/crio-7b3bdffc4bb30a477767be2a0c2e1699eac2d5f3a66269de10f2f1a9f8c234c6 WatchSource:0}: Error finding container 7b3bdffc4bb30a477767be2a0c2e1699eac2d5f3a66269de10f2f1a9f8c234c6: Status 404 returned error can't find the container with id 7b3bdffc4bb30a477767be2a0c2e1699eac2d5f3a66269de10f2f1a9f8c234c6
Jul 24 16:04:53.161389 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:53.161355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v" event={"ID":"82372b79-782b-48d0-b5cc-000500cc6b8b","Type":"ContainerStarted","Data":"b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402"}
Jul 24 16:04:53.161779 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:53.161396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v" event={"ID":"82372b79-782b-48d0-b5cc-000500cc6b8b","Type":"ContainerStarted","Data":"7b3bdffc4bb30a477767be2a0c2e1699eac2d5f3a66269de10f2f1a9f8c234c6"}
Jul 24 16:04:53.180282 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:53.180204    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v" podStartSLOduration=2.180187096 podStartE2EDuration="2.180187096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:04:53.178179413 +0000 UTC m=+1452.662740010" watchObservedRunningTime="2026-07-24 16:04:53.180187096 +0000 UTC m=+1452.664747694"
Jul 24 16:04:56.190380 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:56.190346    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:56.190821 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:56.190584    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:56.203261 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:56.203219    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:04:57.193628 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:04:57.193599    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:05:02.171916 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:02.171881    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:05:02.171916 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:02.171926    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:05:02.184471 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:02.184437    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:05:02.219395 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:02.219369    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:05:04.670628 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:04.670592    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:05:27.694500 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:27.694467    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"]
Jul 24 16:05:28.655700 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:28.655669    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"]
Jul 24 16:05:31.677053 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:31.677017    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:05:41.089123 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:41.089096    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:05:41.093443 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:41.093422    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:05:41.095750 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:41.095721    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:05:41.100272 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:41.100232    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:05:42.714329 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:42.714262    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45" podUID="c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" containerName="main" containerID="cri-o://2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95" gracePeriod=45
Jul 24 16:05:42.967734 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:42.967666    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:05:43.047830 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.047794    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-dshm\") pod \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") "
Jul 24 16:05:43.048016 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.047843    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jss7z\" (UniqueName: \"kubernetes.io/projected/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-kube-api-access-jss7z\") pod \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") "
Jul 24 16:05:43.048016 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.047873    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tls-certs\") pod \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") "
Jul 24 16:05:43.048016 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.047920    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-model-cache\") pod \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") "
Jul 24 16:05:43.048016 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.047965    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-home\") pod \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") "
Jul 24 16:05:43.048016 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.047988    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tmp-dir\") pod \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\" (UID: \"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6\") "
Jul 24 16:05:43.048340 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.048280    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-home" (OuterVolumeSpecName: "home") pod "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" (UID: "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:05:43.048340 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.048286    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" (UID: "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:05:43.048340 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.048324    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" (UID: "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:05:43.050311 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.050284    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" (UID: "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:05:43.050311 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.050294    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-dshm" (OuterVolumeSpecName: "dshm") pod "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" (UID: "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:05:43.050433 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.050324    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-kube-api-access-jss7z" (OuterVolumeSpecName: "kube-api-access-jss7z") pod "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" (UID: "c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6"). InnerVolumeSpecName "kube-api-access-jss7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:05:43.149583 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.149550    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:43.149583 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.149580    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:43.149583 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.149588    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:43.149583 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.149595    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:43.149583 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.149603    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:43.149900 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.149612    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jss7z\" (UniqueName: \"kubernetes.io/projected/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6-kube-api-access-jss7z\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:43.423706 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.423672    2576 generic.go:356] "Generic (PLEG): container finished" podID="c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" containerID="2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95" exitCode=0
Jul 24 16:05:43.423889 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.423712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45" event={"ID":"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6","Type":"ContainerDied","Data":"2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95"}
Jul 24 16:05:43.423889 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.423716    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"
Jul 24 16:05:43.423889 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.423751    2576 scope.go:117] "RemoveContainer" containerID="2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95"
Jul 24 16:05:43.423889 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.423741    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45" event={"ID":"c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6","Type":"ContainerDied","Data":"89a20b0b6542803a3ab4a971f6b1db334646cb5a15e37132f48a5d03ede95aaa"}
Jul 24 16:05:43.433132 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.433111    2576 scope.go:117] "RemoveContainer" containerID="2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95"
Jul 24 16:05:43.433401 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:05:43.433378    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95\": container with ID starting with 2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95 not found: ID does not exist" containerID="2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95"
Jul 24 16:05:43.433479 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.433414    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95"} err="failed to get container status \"2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95\": rpc error: code = NotFound desc = could not find container \"2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95\": container with ID starting with 2f41c25f2141310685c69301d0f22e5c410e588ffe8bf5a6e565fb802c829d95 not found: ID does not exist"
Jul 24 16:05:43.445916 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.445895    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"]
Jul 24 16:05:43.449382 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.449362    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5cd8d52e-kserve-7d95c55999-97s45"]
Jul 24 16:05:43.675656 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.675564    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v" podUID="82372b79-782b-48d0-b5cc-000500cc6b8b" containerName="main" containerID="cri-o://b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402" gracePeriod=45
Jul 24 16:05:43.927174 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:43.927116    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:05:44.057482 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057447    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-home\") pod \"82372b79-782b-48d0-b5cc-000500cc6b8b\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") "
Jul 24 16:05:44.057653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057500    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82372b79-782b-48d0-b5cc-000500cc6b8b-tls-certs\") pod \"82372b79-782b-48d0-b5cc-000500cc6b8b\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") "
Jul 24 16:05:44.057653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057562    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-tmp-dir\") pod \"82372b79-782b-48d0-b5cc-000500cc6b8b\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") "
Jul 24 16:05:44.057653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057584    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-model-cache\") pod \"82372b79-782b-48d0-b5cc-000500cc6b8b\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") "
Jul 24 16:05:44.057653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057611    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-dshm\") pod \"82372b79-782b-48d0-b5cc-000500cc6b8b\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") "
Jul 24 16:05:44.057653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057649    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkgnl\" (UniqueName: \"kubernetes.io/projected/82372b79-782b-48d0-b5cc-000500cc6b8b-kube-api-access-nkgnl\") pod \"82372b79-782b-48d0-b5cc-000500cc6b8b\" (UID: \"82372b79-782b-48d0-b5cc-000500cc6b8b\") "
Jul 24 16:05:44.057849 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057708    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-home" (OuterVolumeSpecName: "home") pod "82372b79-782b-48d0-b5cc-000500cc6b8b" (UID: "82372b79-782b-48d0-b5cc-000500cc6b8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:05:44.057849 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057814    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "82372b79-782b-48d0-b5cc-000500cc6b8b" (UID: "82372b79-782b-48d0-b5cc-000500cc6b8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:05:44.057849 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057829    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82372b79-782b-48d0-b5cc-000500cc6b8b" (UID: "82372b79-782b-48d0-b5cc-000500cc6b8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:05:44.057974 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057953    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:44.058038 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057977    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:44.058038 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.057986    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:44.059973 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.059949    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82372b79-782b-48d0-b5cc-000500cc6b8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82372b79-782b-48d0-b5cc-000500cc6b8b" (UID: "82372b79-782b-48d0-b5cc-000500cc6b8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:05:44.059973 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.059965    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-dshm" (OuterVolumeSpecName: "dshm") pod "82372b79-782b-48d0-b5cc-000500cc6b8b" (UID: "82372b79-782b-48d0-b5cc-000500cc6b8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:05:44.060126 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.060034    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82372b79-782b-48d0-b5cc-000500cc6b8b-kube-api-access-nkgnl" (OuterVolumeSpecName: "kube-api-access-nkgnl") pod "82372b79-782b-48d0-b5cc-000500cc6b8b" (UID: "82372b79-782b-48d0-b5cc-000500cc6b8b"). InnerVolumeSpecName "kube-api-access-nkgnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:05:44.158435 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.158404    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82372b79-782b-48d0-b5cc-000500cc6b8b-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:44.158435 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.158432    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82372b79-782b-48d0-b5cc-000500cc6b8b-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:44.158435 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.158442    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgnl\" (UniqueName: \"kubernetes.io/projected/82372b79-782b-48d0-b5cc-000500cc6b8b-kube-api-access-nkgnl\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:05:44.172918 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.172887    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:05:44.431294 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.431259    2576 generic.go:356] "Generic (PLEG): container finished" podID="82372b79-782b-48d0-b5cc-000500cc6b8b" containerID="b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402" exitCode=0
Jul 24 16:05:44.431294 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.431302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v" event={"ID":"82372b79-782b-48d0-b5cc-000500cc6b8b","Type":"ContainerDied","Data":"b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402"}
Jul 24 16:05:44.431537 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.431313    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"
Jul 24 16:05:44.431537 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.431337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v" event={"ID":"82372b79-782b-48d0-b5cc-000500cc6b8b","Type":"ContainerDied","Data":"7b3bdffc4bb30a477767be2a0c2e1699eac2d5f3a66269de10f2f1a9f8c234c6"}
Jul 24 16:05:44.431537 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.431357    2576 scope.go:117] "RemoveContainer" containerID="b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402"
Jul 24 16:05:44.440840 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.440826    2576 scope.go:117] "RemoveContainer" containerID="b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402"
Jul 24 16:05:44.441105 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:05:44.441085    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402\": container with ID starting with b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402 not found: ID does not exist" containerID="b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402"
Jul 24 16:05:44.441202 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.441113    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402"} err="failed to get container status \"b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402\": rpc error: code = NotFound desc = could not find container \"b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402\": container with ID starting with b06dcc837224e4eef34631b04815c42bc335572b2c931dc675d8522ebebea402 not found: ID does not exist"
Jul 24 16:05:44.452832 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.452806    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"]
Jul 24 16:05:44.458602 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:44.458569    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5cd8d52e-kserve-867c4cdfbc-hc64v"]
Jul 24 16:05:45.005423 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:45.005389    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82372b79-782b-48d0-b5cc-000500cc6b8b" path="/var/lib/kubelet/pods/82372b79-782b-48d0-b5cc-000500cc6b8b/volumes"
Jul 24 16:05:45.005788 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:45.005775    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" path="/var/lib/kubelet/pods/c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6/volumes"
Jul 24 16:05:54.667522 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:05:54.667485    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:06:06.073108 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:06:06.073066    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:06:17.173137 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:06:17.173100    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:06:26.277411 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:06:26.277368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:06:35.265123 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:06:35.265075    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:06:45.871617 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:06:45.871572    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:06:55.573535 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:06:55.573500    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:07:08.565985 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:07:08.565950    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:07:17.371808 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:07:17.371773    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:07:27.469319 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:07:27.469281    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:07:36.870845 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:07:36.870763    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:07:50.265145 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:07:50.265110    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:07:57.764583 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:07:57.764544    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:08:09.869927 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:08:09.869888    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:08:21.577457 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:08:21.577415    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:08:33.166845 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:08:33.166803    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:08:45.977348 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:08:45.977314    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:09:00.964097 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:09:00.964060    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:09:16.372046 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:09:16.372006    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:09:48.268053 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:09:48.268014    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:09:59.767034 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:09:59.766998    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:10:31.686661 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:10:31.686611    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:10:41.122044 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:10:41.122015    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:10:41.127192 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:10:41.127167    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:10:41.131505 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:10:41.131487    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:10:41.135610 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:10:41.135593    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:10:42.771972 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:10:42.771931    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:11:15.867083 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:11:15.867051    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:11:27.263317 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:11:27.263284    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:12:00.867128 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:12:00.867094    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:13:27.164411 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:13:27.164367    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:13:37.863260 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:13:37.863153    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:13:47.865816 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:13:47.865784    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:14:08.066172 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:08.066134    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:14:46.565863 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:46.565827    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:14:52.274275 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.274225    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"]
Jul 24 16:14:52.274764 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.274742    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82372b79-782b-48d0-b5cc-000500cc6b8b" containerName="main"
Jul 24 16:14:52.274836 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.274767    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82372b79-782b-48d0-b5cc-000500cc6b8b" containerName="main"
Jul 24 16:14:52.274836 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.274799    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" containerName="main"
Jul 24 16:14:52.274836 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.274809    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" containerName="main"
Jul 24 16:14:52.274992 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.274919    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8dc82bc-6dc9-42bc-9bcc-1bd26a43b8f6" containerName="main"
Jul 24 16:14:52.274992 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.274932    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82372b79-782b-48d0-b5cc-000500cc6b8b" containerName="main"
Jul 24 16:14:52.277728 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.277707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.280559 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.280532    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b89f991b\"/\"openshift-service-ca.crt\""
Jul 24 16:14:52.281641 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.281622    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b89f991b\"/\"default-dockercfg-p9q8k\""
Jul 24 16:14:52.281748 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.281645    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b89f991b\"/\"kube-root-ca.crt\""
Jul 24 16:14:52.281802 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.281785    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b89f991b\"/\"e2e-lifecycle-model-b89f991b-kserve-self-signed-certs\""
Jul 24 16:14:52.285845 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.285825    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"]
Jul 24 16:14:52.418181 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.418150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50405012-853d-4143-864c-7e959531b8b0-tls-certs\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.418368 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.418200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-home\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.418368 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.418275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-dshm\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.418368 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.418321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-tmp-dir\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.418368 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.418345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-model-cache\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.418368 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.418363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcz7m\" (UniqueName: \"kubernetes.io/projected/50405012-853d-4143-864c-7e959531b8b0-kube-api-access-tcz7m\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.519414 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.519388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-home\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.519591 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.519422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-dshm\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.519591 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.519451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-tmp-dir\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.519591 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.519470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-model-cache\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.519591 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.519489    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcz7m\" (UniqueName: \"kubernetes.io/projected/50405012-853d-4143-864c-7e959531b8b0-kube-api-access-tcz7m\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.519786 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.519601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50405012-853d-4143-864c-7e959531b8b0-tls-certs\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.519899 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.519876    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-tmp-dir\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.519966 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.519893    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-home\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.519966 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.519940    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-model-cache\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.521808 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.521785    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-dshm\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.522053 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.522036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50405012-853d-4143-864c-7e959531b8b0-tls-certs\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.527924 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.527866    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcz7m\" (UniqueName: \"kubernetes.io/projected/50405012-853d-4143-864c-7e959531b8b0-kube-api-access-tcz7m\") pod \"e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") " pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.588599 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.588568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:14:52.721650 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:52.721620    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"]
Jul 24 16:14:52.723349 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:14:52.723321    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50405012_853d_4143_864c_7e959531b8b0.slice/crio-fffc23ffb181ba711bb87e50aa6db7031f458ce7c0ed5ef30b769d2da2a2628c WatchSource:0}: Error finding container fffc23ffb181ba711bb87e50aa6db7031f458ce7c0ed5ef30b769d2da2a2628c: Status 404 returned error can't find the container with id fffc23ffb181ba711bb87e50aa6db7031f458ce7c0ed5ef30b769d2da2a2628c
Jul 24 16:14:53.204796 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:53.204757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px" event={"ID":"50405012-853d-4143-864c-7e959531b8b0","Type":"ContainerStarted","Data":"f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0"}
Jul 24 16:14:53.204972 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:53.204809    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px" event={"ID":"50405012-853d-4143-864c-7e959531b8b0","Type":"ContainerStarted","Data":"fffc23ffb181ba711bb87e50aa6db7031f458ce7c0ed5ef30b769d2da2a2628c"}
Jul 24 16:14:53.223970 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:53.223920    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px" podStartSLOduration=1.223901961 podStartE2EDuration="1.223901961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:14:53.221617243 +0000 UTC m=+2052.706177845" watchObservedRunningTime="2026-07-24 16:14:53.223901961 +0000 UTC m=+2052.708462559"
Jul 24 16:14:57.967267 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:14:57.967217    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:15:02.589518 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:02.589423    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:15:02.589518 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:02.589473    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:15:02.602125 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:02.602100    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:15:03.264026 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:03.263995    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:15:21.469134 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:21.469094    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:15:41.156096 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:41.156062    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:15:41.160623 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:41.160594    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:15:41.164872 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:41.164851    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:15:41.168856 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:41.168841    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:15:42.591770 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:42.591739    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:15:53.560146 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:53.560106    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:15:56.955497 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:15:56.955455    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"]
Jul 24 16:16:11.975031 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:11.974957    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px" podUID="50405012-853d-4143-864c-7e959531b8b0" containerName="main" containerID="cri-o://f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0" gracePeriod=45
Jul 24 16:16:12.224078 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.224055    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:16:12.394571 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.394539    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-tmp-dir\") pod \"50405012-853d-4143-864c-7e959531b8b0\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") "
Jul 24 16:16:12.394735 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.394590    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-home\") pod \"50405012-853d-4143-864c-7e959531b8b0\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") "
Jul 24 16:16:12.394735 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.394628    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcz7m\" (UniqueName: \"kubernetes.io/projected/50405012-853d-4143-864c-7e959531b8b0-kube-api-access-tcz7m\") pod \"50405012-853d-4143-864c-7e959531b8b0\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") "
Jul 24 16:16:12.394735 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.394663    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-dshm\") pod \"50405012-853d-4143-864c-7e959531b8b0\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") "
Jul 24 16:16:12.394735 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.394693    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50405012-853d-4143-864c-7e959531b8b0-tls-certs\") pod \"50405012-853d-4143-864c-7e959531b8b0\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") "
Jul 24 16:16:12.394944 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.394756    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-model-cache\") pod \"50405012-853d-4143-864c-7e959531b8b0\" (UID: \"50405012-853d-4143-864c-7e959531b8b0\") "
Jul 24 16:16:12.394944 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.394882    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-home" (OuterVolumeSpecName: "home") pod "50405012-853d-4143-864c-7e959531b8b0" (UID: "50405012-853d-4143-864c-7e959531b8b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:16:12.395083 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.395030    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:16:12.395083 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.395040    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50405012-853d-4143-864c-7e959531b8b0" (UID: "50405012-853d-4143-864c-7e959531b8b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:16:12.395262 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.395095    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "50405012-853d-4143-864c-7e959531b8b0" (UID: "50405012-853d-4143-864c-7e959531b8b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:16:12.397501 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.397466    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-dshm" (OuterVolumeSpecName: "dshm") pod "50405012-853d-4143-864c-7e959531b8b0" (UID: "50405012-853d-4143-864c-7e959531b8b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:16:12.397589 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.397528    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50405012-853d-4143-864c-7e959531b8b0-kube-api-access-tcz7m" (OuterVolumeSpecName: "kube-api-access-tcz7m") pod "50405012-853d-4143-864c-7e959531b8b0" (UID: "50405012-853d-4143-864c-7e959531b8b0"). InnerVolumeSpecName "kube-api-access-tcz7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:16:12.397589 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.397528    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50405012-853d-4143-864c-7e959531b8b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50405012-853d-4143-864c-7e959531b8b0" (UID: "50405012-853d-4143-864c-7e959531b8b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:16:12.495622 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.495584    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:16:12.495622 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.495617    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcz7m\" (UniqueName: \"kubernetes.io/projected/50405012-853d-4143-864c-7e959531b8b0-kube-api-access-tcz7m\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:16:12.495622 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.495629    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:16:12.495874 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.495641    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50405012-853d-4143-864c-7e959531b8b0-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:16:12.495874 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.495652    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50405012-853d-4143-864c-7e959531b8b0-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:16:12.605778 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.605735    2576 generic.go:356] "Generic (PLEG): container finished" podID="50405012-853d-4143-864c-7e959531b8b0" containerID="f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0" exitCode=0
Jul 24 16:16:12.605952 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.605792    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"
Jul 24 16:16:12.605952 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.605826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px" event={"ID":"50405012-853d-4143-864c-7e959531b8b0","Type":"ContainerDied","Data":"f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0"}
Jul 24 16:16:12.605952 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.605875    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px" event={"ID":"50405012-853d-4143-864c-7e959531b8b0","Type":"ContainerDied","Data":"fffc23ffb181ba711bb87e50aa6db7031f458ce7c0ed5ef30b769d2da2a2628c"}
Jul 24 16:16:12.605952 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.605896    2576 scope.go:117] "RemoveContainer" containerID="f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0"
Jul 24 16:16:12.615325 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.615311    2576 scope.go:117] "RemoveContainer" containerID="f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0"
Jul 24 16:16:12.615588 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:16:12.615568    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0\": container with ID starting with f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0 not found: ID does not exist" containerID="f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0"
Jul 24 16:16:12.615637 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.615596    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0"} err="failed to get container status \"f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0\": rpc error: code = NotFound desc = could not find container \"f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0\": container with ID starting with f53d8e55b180dd5bffc0d2542dbf83b1fa9fd4ddc51a79652e3bcfac4c9f78d0 not found: ID does not exist"
Jul 24 16:16:12.628988 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.628960    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"]
Jul 24 16:16:12.631045 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:12.631026    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b89f991b/e2e-lifecycle-model-b89f991b-kserve-779ccfffd7-m62px"]
Jul 24 16:16:13.005280 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:13.005223    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50405012-853d-4143-864c-7e959531b8b0" path="/var/lib/kubelet/pods/50405012-853d-4143-864c-7e959531b8b0/volumes"
Jul 24 16:16:43.673572 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:43.673533    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:16:56.180894 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:16:56.180856    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:17:44.568262 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:17:44.568201    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:18:01.472185 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.472142    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:18:01.585332 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.585299    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"]
Jul 24 16:18:01.585633 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.585621    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50405012-853d-4143-864c-7e959531b8b0" containerName="main"
Jul 24 16:18:01.585678 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.585634    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50405012-853d-4143-864c-7e959531b8b0" containerName="main"
Jul 24 16:18:01.585718 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.585699    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50405012-853d-4143-864c-7e959531b8b0" containerName="main"
Jul 24 16:18:01.588580 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.588564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.591664 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.591644    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-35cc2d\"/\"openshift-service-ca.crt\""
Jul 24 16:18:01.592611 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.592594    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-35cc2d\"/\"default-dockercfg-76nq5\""
Jul 24 16:18:01.592683 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.592664    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-35cc2d\"/\"kube-root-ca.crt\""
Jul 24 16:18:01.592784 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.592773    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-35cc2d\"/\"test-model-35cc2d-kserve-self-signed-certs\""
Jul 24 16:18:01.606482 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.606452    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"]
Jul 24 16:18:01.729631 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.729541    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-home\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.729631 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.729583    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-model-cache\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.729631 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.729607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5q89\" (UniqueName: \"kubernetes.io/projected/a210f5db-6026-4cda-91ef-a39a993552f2-kube-api-access-b5q89\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.729849 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.729692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a210f5db-6026-4cda-91ef-a39a993552f2-tls-certs\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.729849 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.729730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-dshm\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.729849 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.729754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-tmp-dir\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.830935 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.830897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-home\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.831142 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.830941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-model-cache\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.831142 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.830970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5q89\" (UniqueName: \"kubernetes.io/projected/a210f5db-6026-4cda-91ef-a39a993552f2-kube-api-access-b5q89\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.831142 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.831004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a210f5db-6026-4cda-91ef-a39a993552f2-tls-certs\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.831142 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.831030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-dshm\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.831142 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.831058    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-tmp-dir\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.831541 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.831518    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-model-cache\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.831616 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.831591    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-tmp-dir\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.831653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.831629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-home\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.833423 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.833405    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-dshm\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.833773 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.833757    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a210f5db-6026-4cda-91ef-a39a993552f2-tls-certs\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.839816 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.839796    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5q89\" (UniqueName: \"kubernetes.io/projected/a210f5db-6026-4cda-91ef-a39a993552f2-kube-api-access-b5q89\") pod \"test-model-35cc2d-kserve-7bc8b8747c-5slw6\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") " pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:01.897543 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:01.897513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:02.032088 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:02.032060    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"]
Jul 24 16:18:02.033701 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:18:02.033674    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda210f5db_6026_4cda_91ef_a39a993552f2.slice/crio-c427d30a4c418f5ab720099dfe6ce28967a5be23b5636707b70edc6286f72ac6 WatchSource:0}: Error finding container c427d30a4c418f5ab720099dfe6ce28967a5be23b5636707b70edc6286f72ac6: Status 404 returned error can't find the container with id c427d30a4c418f5ab720099dfe6ce28967a5be23b5636707b70edc6286f72ac6
Jul 24 16:18:02.157093 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:02.157009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6" event={"ID":"a210f5db-6026-4cda-91ef-a39a993552f2","Type":"ContainerStarted","Data":"a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760"}
Jul 24 16:18:02.157093 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:02.157051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6" event={"ID":"a210f5db-6026-4cda-91ef-a39a993552f2","Type":"ContainerStarted","Data":"c427d30a4c418f5ab720099dfe6ce28967a5be23b5636707b70edc6286f72ac6"}
Jul 24 16:18:02.175202 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:02.175154    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6" podStartSLOduration=1.175138521 podStartE2EDuration="1.175138521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:18:02.174496806 +0000 UTC m=+2241.659057419" watchObservedRunningTime="2026-07-24 16:18:02.175138521 +0000 UTC m=+2241.659699119"
Jul 24 16:18:11.897922 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:11.897881    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:11.898352 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:11.897975    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:11.910373 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:11.910348    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:12.218299 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:12.218214    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:18:12.371486 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:12.371442    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:18:24.973737 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:24.973702    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"]
Jul 24 16:18:24.977628 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:24.977608    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:24.981489 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:24.981457    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-47a0d8\"/\"kube-root-ca.crt\""
Jul 24 16:18:24.981489 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:24.981457    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-47a0d8\"/\"default-dockercfg-djh7p\""
Jul 24 16:18:24.981804 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:24.981468    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-47a0d8\"/\"test-model-47a0d8-kserve-self-signed-certs\""
Jul 24 16:18:24.981804 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:24.981468    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-47a0d8\"/\"openshift-service-ca.crt\""
Jul 24 16:18:24.984307 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:24.984283    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"]
Jul 24 16:18:25.027025 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.026997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-model-cache\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.027025 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.027028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d79a273-da33-454c-9723-bb33c5c9a53f-tls-certs\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.027303 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.027117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-tmp-dir\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.027303 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.027156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-dshm\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.027418 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.027297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-home\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.027418 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.027353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbgm\" (UniqueName: \"kubernetes.io/projected/8d79a273-da33-454c-9723-bb33c5c9a53f-kube-api-access-jhbgm\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.128235 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.128199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbgm\" (UniqueName: \"kubernetes.io/projected/8d79a273-da33-454c-9723-bb33c5c9a53f-kube-api-access-jhbgm\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.128235 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.128262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-model-cache\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.128524 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.128305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d79a273-da33-454c-9723-bb33c5c9a53f-tls-certs\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.128524 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.128368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-tmp-dir\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.128524 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.128408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-dshm\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.128524 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.128486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-home\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.128745 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.128617    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-model-cache\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.128804 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.128745    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-tmp-dir\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.128858 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.128842    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-home\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.130843 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.130821    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-dshm\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.131129 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.131108    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d79a273-da33-454c-9723-bb33c5c9a53f-tls-certs\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.136682 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.136658    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbgm\" (UniqueName: \"kubernetes.io/projected/8d79a273-da33-454c-9723-bb33c5c9a53f-kube-api-access-jhbgm\") pod \"test-model-47a0d8-kserve-744f8d47fc-fljsn\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") " pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.288853 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.288766    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:25.426901 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:25.426878    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"]
Jul 24 16:18:25.428979 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:18:25.428952    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d79a273_da33_454c_9723_bb33c5c9a53f.slice/crio-2a253584c9e3d1fa610891087b73b8af00bc31d3f408d10076236779fad015f8 WatchSource:0}: Error finding container 2a253584c9e3d1fa610891087b73b8af00bc31d3f408d10076236779fad015f8: Status 404 returned error can't find the container with id 2a253584c9e3d1fa610891087b73b8af00bc31d3f408d10076236779fad015f8
Jul 24 16:18:26.281270 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:26.281213    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn" event={"ID":"8d79a273-da33-454c-9723-bb33c5c9a53f","Type":"ContainerStarted","Data":"3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e"}
Jul 24 16:18:26.281270 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:26.281275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn" event={"ID":"8d79a273-da33-454c-9723-bb33c5c9a53f","Type":"ContainerStarted","Data":"2a253584c9e3d1fa610891087b73b8af00bc31d3f408d10076236779fad015f8"}
Jul 24 16:18:26.299102 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:26.299056    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn" podStartSLOduration=2.299042591 podStartE2EDuration="2.299042591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:18:26.297624411 +0000 UTC m=+2265.782185034" watchObservedRunningTime="2026-07-24 16:18:26.299042591 +0000 UTC m=+2265.783603189"
Jul 24 16:18:29.769449 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:29.769413    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:18:35.288879 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:35.288835    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:35.289386 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:35.288896    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:35.301705 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:35.301681    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:35.338454 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:35.338430    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:18:43.467394 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:18:43.467355    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:19:05.558325 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:05.558290    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"]
Jul 24 16:19:06.545339 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:06.545302    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"]
Jul 24 16:19:12.174043 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:12.174008    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:19:20.166414 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.166378    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:19:20.577679 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.577568    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6" podUID="a210f5db-6026-4cda-91ef-a39a993552f2" containerName="main" containerID="cri-o://a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760" gracePeriod=45
Jul 24 16:19:20.827542 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.827520    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:19:20.918964 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.918932    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-model-cache\") pod \"a210f5db-6026-4cda-91ef-a39a993552f2\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") "
Jul 24 16:19:20.919132 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919021    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-home\") pod \"a210f5db-6026-4cda-91ef-a39a993552f2\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") "
Jul 24 16:19:20.919132 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919059    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-tmp-dir\") pod \"a210f5db-6026-4cda-91ef-a39a993552f2\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") "
Jul 24 16:19:20.919132 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919102    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a210f5db-6026-4cda-91ef-a39a993552f2-tls-certs\") pod \"a210f5db-6026-4cda-91ef-a39a993552f2\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") "
Jul 24 16:19:20.919132 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919125    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5q89\" (UniqueName: \"kubernetes.io/projected/a210f5db-6026-4cda-91ef-a39a993552f2-kube-api-access-b5q89\") pod \"a210f5db-6026-4cda-91ef-a39a993552f2\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") "
Jul 24 16:19:20.919361 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-dshm\") pod \"a210f5db-6026-4cda-91ef-a39a993552f2\" (UID: \"a210f5db-6026-4cda-91ef-a39a993552f2\") "
Jul 24 16:19:20.919361 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919213    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a210f5db-6026-4cda-91ef-a39a993552f2" (UID: "a210f5db-6026-4cda-91ef-a39a993552f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:19:20.919361 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919288    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-home" (OuterVolumeSpecName: "home") pod "a210f5db-6026-4cda-91ef-a39a993552f2" (UID: "a210f5db-6026-4cda-91ef-a39a993552f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:19:20.919361 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919322    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a210f5db-6026-4cda-91ef-a39a993552f2" (UID: "a210f5db-6026-4cda-91ef-a39a993552f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:19:20.919513 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919476    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:20.919513 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919496    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:20.919580 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.919512    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:20.921579 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.921553    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a210f5db-6026-4cda-91ef-a39a993552f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a210f5db-6026-4cda-91ef-a39a993552f2" (UID: "a210f5db-6026-4cda-91ef-a39a993552f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:19:20.921668 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.921580    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a210f5db-6026-4cda-91ef-a39a993552f2-kube-api-access-b5q89" (OuterVolumeSpecName: "kube-api-access-b5q89") pod "a210f5db-6026-4cda-91ef-a39a993552f2" (UID: "a210f5db-6026-4cda-91ef-a39a993552f2"). InnerVolumeSpecName "kube-api-access-b5q89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:19:20.921668 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:20.921591    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-dshm" (OuterVolumeSpecName: "dshm") pod "a210f5db-6026-4cda-91ef-a39a993552f2" (UID: "a210f5db-6026-4cda-91ef-a39a993552f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:19:21.020506 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.020480    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a210f5db-6026-4cda-91ef-a39a993552f2-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:21.020506 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.020505    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5q89\" (UniqueName: \"kubernetes.io/projected/a210f5db-6026-4cda-91ef-a39a993552f2-kube-api-access-b5q89\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:21.020671 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.020515    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a210f5db-6026-4cda-91ef-a39a993552f2-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:21.564763 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.564719    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn" podUID="8d79a273-da33-454c-9723-bb33c5c9a53f" containerName="main" containerID="cri-o://3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e" gracePeriod=45
Jul 24 16:19:21.565475 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.565451    2576 generic.go:356] "Generic (PLEG): container finished" podID="a210f5db-6026-4cda-91ef-a39a993552f2" containerID="a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760" exitCode=0
Jul 24 16:19:21.565545 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.565511    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"
Jul 24 16:19:21.565602 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.565540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6" event={"ID":"a210f5db-6026-4cda-91ef-a39a993552f2","Type":"ContainerDied","Data":"a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760"}
Jul 24 16:19:21.565602 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.565594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6" event={"ID":"a210f5db-6026-4cda-91ef-a39a993552f2","Type":"ContainerDied","Data":"c427d30a4c418f5ab720099dfe6ce28967a5be23b5636707b70edc6286f72ac6"}
Jul 24 16:19:21.565704 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.565624    2576 scope.go:117] "RemoveContainer" containerID="a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760"
Jul 24 16:19:21.586488 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.586467    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"]
Jul 24 16:19:21.590485 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.590465    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-35cc2d/test-model-35cc2d-kserve-7bc8b8747c-5slw6"]
Jul 24 16:19:21.623098 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.622947    2576 scope.go:117] "RemoveContainer" containerID="a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760"
Jul 24 16:19:21.623352 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:19:21.623332    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760\": container with ID starting with a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760 not found: ID does not exist" containerID="a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760"
Jul 24 16:19:21.623399 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.623362    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760"} err="failed to get container status \"a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760\": rpc error: code = NotFound desc = could not find container \"a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760\": container with ID starting with a2ebd5c6d94913c9b8cd0d9fdb741324954c4ca0d99d5cb91767330f33bce760 not found: ID does not exist"
Jul 24 16:19:21.812015 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.811993    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:19:21.928226 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928203    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-tmp-dir\") pod \"8d79a273-da33-454c-9723-bb33c5c9a53f\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") "
Jul 24 16:19:21.928378 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928257    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-dshm\") pod \"8d79a273-da33-454c-9723-bb33c5c9a53f\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") "
Jul 24 16:19:21.928378 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928293    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-model-cache\") pod \"8d79a273-da33-454c-9723-bb33c5c9a53f\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") "
Jul 24 16:19:21.928378 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928340    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhbgm\" (UniqueName: \"kubernetes.io/projected/8d79a273-da33-454c-9723-bb33c5c9a53f-kube-api-access-jhbgm\") pod \"8d79a273-da33-454c-9723-bb33c5c9a53f\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") "
Jul 24 16:19:21.928378 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928367    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d79a273-da33-454c-9723-bb33c5c9a53f-tls-certs\") pod \"8d79a273-da33-454c-9723-bb33c5c9a53f\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") "
Jul 24 16:19:21.928591 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928397    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-home\") pod \"8d79a273-da33-454c-9723-bb33c5c9a53f\" (UID: \"8d79a273-da33-454c-9723-bb33c5c9a53f\") "
Jul 24 16:19:21.928591 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928497    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d79a273-da33-454c-9723-bb33c5c9a53f" (UID: "8d79a273-da33-454c-9723-bb33c5c9a53f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:19:21.928591 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928551    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d79a273-da33-454c-9723-bb33c5c9a53f" (UID: "8d79a273-da33-454c-9723-bb33c5c9a53f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:19:21.928748 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928654    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:21.928748 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928673    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:21.928748 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.928688    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-home" (OuterVolumeSpecName: "home") pod "8d79a273-da33-454c-9723-bb33c5c9a53f" (UID: "8d79a273-da33-454c-9723-bb33c5c9a53f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:19:21.930655 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.930617    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-dshm" (OuterVolumeSpecName: "dshm") pod "8d79a273-da33-454c-9723-bb33c5c9a53f" (UID: "8d79a273-da33-454c-9723-bb33c5c9a53f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:19:21.930765 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.930687    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d79a273-da33-454c-9723-bb33c5c9a53f-kube-api-access-jhbgm" (OuterVolumeSpecName: "kube-api-access-jhbgm") pod "8d79a273-da33-454c-9723-bb33c5c9a53f" (UID: "8d79a273-da33-454c-9723-bb33c5c9a53f"). InnerVolumeSpecName "kube-api-access-jhbgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:19:21.930765 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:21.930695    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d79a273-da33-454c-9723-bb33c5c9a53f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d79a273-da33-454c-9723-bb33c5c9a53f" (UID: "8d79a273-da33-454c-9723-bb33c5c9a53f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:19:22.029333 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.029310    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:22.029333 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.029331    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbgm\" (UniqueName: \"kubernetes.io/projected/8d79a273-da33-454c-9723-bb33c5c9a53f-kube-api-access-jhbgm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:22.029453 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.029340    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d79a273-da33-454c-9723-bb33c5c9a53f-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:22.029453 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.029349    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d79a273-da33-454c-9723-bb33c5c9a53f-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:19:22.572883 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.572845    2576 generic.go:356] "Generic (PLEG): container finished" podID="8d79a273-da33-454c-9723-bb33c5c9a53f" containerID="3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e" exitCode=0
Jul 24 16:19:22.572883 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.572884    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"
Jul 24 16:19:22.573473 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.572928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn" event={"ID":"8d79a273-da33-454c-9723-bb33c5c9a53f","Type":"ContainerDied","Data":"3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e"}
Jul 24 16:19:22.573473 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.572969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn" event={"ID":"8d79a273-da33-454c-9723-bb33c5c9a53f","Type":"ContainerDied","Data":"2a253584c9e3d1fa610891087b73b8af00bc31d3f408d10076236779fad015f8"}
Jul 24 16:19:22.573473 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.572994    2576 scope.go:117] "RemoveContainer" containerID="3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e"
Jul 24 16:19:22.582906 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.582890    2576 scope.go:117] "RemoveContainer" containerID="3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e"
Jul 24 16:19:22.583165 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:19:22.583146    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e\": container with ID starting with 3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e not found: ID does not exist" containerID="3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e"
Jul 24 16:19:22.583265 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.583173    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e"} err="failed to get container status \"3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e\": rpc error: code = NotFound desc = could not find container \"3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e\": container with ID starting with 3915d2862c1a28aa9990beba51dd1bb35d35f8af14127bcd8ece00ae6fb91b2e not found: ID does not exist"
Jul 24 16:19:22.606055 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.606021    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"]
Jul 24 16:19:22.612493 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:22.612468    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-47a0d8/test-model-47a0d8-kserve-744f8d47fc-fljsn"]
Jul 24 16:19:23.007212 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:23.007170    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d79a273-da33-454c-9723-bb33c5c9a53f" path="/var/lib/kubelet/pods/8d79a273-da33-454c-9723-bb33c5c9a53f/volumes"
Jul 24 16:19:23.010546 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:23.007831    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a210f5db-6026-4cda-91ef-a39a993552f2" path="/var/lib/kubelet/pods/a210f5db-6026-4cda-91ef-a39a993552f2/volumes"
Jul 24 16:19:42.267553 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:19:42.267514    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:20:02.068408 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:20:02.068364    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:20:18.174322 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:20:18.174285    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:20:34.068089 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:20:34.068051    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:20:41.188062 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:20:41.188031    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:20:41.192092 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:20:41.192067    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:20:41.197723 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:20:41.197703    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:20:41.201681 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:20:41.201662    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:21:02.076724 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:02.076686    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:21:16.475998 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:16.475962    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:21:34.164182 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:34.164150    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:21:46.072738 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.072705    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"]
Jul 24 16:21:46.073103 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.073086    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a210f5db-6026-4cda-91ef-a39a993552f2" containerName="main"
Jul 24 16:21:46.073103 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.073098    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a210f5db-6026-4cda-91ef-a39a993552f2" containerName="main"
Jul 24 16:21:46.073179 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.073111    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d79a273-da33-454c-9723-bb33c5c9a53f" containerName="main"
Jul 24 16:21:46.073179 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.073118    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d79a273-da33-454c-9723-bb33c5c9a53f" containerName="main"
Jul 24 16:21:46.073260 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.073181    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d79a273-da33-454c-9723-bb33c5c9a53f" containerName="main"
Jul 24 16:21:46.073260 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.073191    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a210f5db-6026-4cda-91ef-a39a993552f2" containerName="main"
Jul 24 16:21:46.076346 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.076330    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.080221 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.080199    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ed0643\"/\"kube-root-ca.crt\""
Jul 24 16:21:46.080221 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.080209    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ed0643\"/\"openshift-service-ca.crt\""
Jul 24 16:21:46.080451 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.080221    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ed0643\"/\"auth-test-model-ed0643-kserve-self-signed-certs\""
Jul 24 16:21:46.080451 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.080221    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ed0643\"/\"default-dockercfg-n6jxn\""
Jul 24 16:21:46.084940 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.084915    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"]
Jul 24 16:21:46.262525 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.262487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-home\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.262718 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.262530    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-dshm\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.262718 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.262592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrfp\" (UniqueName: \"kubernetes.io/projected/05005c72-dfa3-499a-af4e-85af8d4d3118-kube-api-access-tnrfp\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.262718 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.262647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05005c72-dfa3-499a-af4e-85af8d4d3118-tls-certs\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.262718 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.262679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-model-cache\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.262718 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.262696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-tmp-dir\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.363524 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.363429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-home\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.363524 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.363469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-dshm\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.363524 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.363503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrfp\" (UniqueName: \"kubernetes.io/projected/05005c72-dfa3-499a-af4e-85af8d4d3118-kube-api-access-tnrfp\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.363786 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.363539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05005c72-dfa3-499a-af4e-85af8d4d3118-tls-certs\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.363786 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.363573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-model-cache\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.363786 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.363588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-tmp-dir\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.363958 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.363859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-home\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.363958 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.363911    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-tmp-dir\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.364063 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.363991    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-model-cache\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.365958 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.365934    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-dshm\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.366212 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.366193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05005c72-dfa3-499a-af4e-85af8d4d3118-tls-certs\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.372297 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.372273    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrfp\" (UniqueName: \"kubernetes.io/projected/05005c72-dfa3-499a-af4e-85af8d4d3118-kube-api-access-tnrfp\") pod \"auth-test-model-ed0643-kserve-8547d85d8c-hvlqq\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") " pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.386335 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.386304    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:46.527805 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:46.527775    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"]
Jul 24 16:21:46.528905 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:21:46.528878    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05005c72_dfa3_499a_af4e_85af8d4d3118.slice/crio-69db1b479ecf051673152653b51100b5b0d99da34f2ec6b13502e3e4d4d4b5d3 WatchSource:0}: Error finding container 69db1b479ecf051673152653b51100b5b0d99da34f2ec6b13502e3e4d4d4b5d3: Status 404 returned error can't find the container with id 69db1b479ecf051673152653b51100b5b0d99da34f2ec6b13502e3e4d4d4b5d3
Jul 24 16:21:47.311040 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:47.310999    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq" event={"ID":"05005c72-dfa3-499a-af4e-85af8d4d3118","Type":"ContainerStarted","Data":"0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7"}
Jul 24 16:21:47.311040 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:47.311042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq" event={"ID":"05005c72-dfa3-499a-af4e-85af8d4d3118","Type":"ContainerStarted","Data":"69db1b479ecf051673152653b51100b5b0d99da34f2ec6b13502e3e4d4d4b5d3"}
Jul 24 16:21:47.331848 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:47.331799    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq" podStartSLOduration=1.331782706 podStartE2EDuration="1.331782706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:21:47.329913704 +0000 UTC m=+2466.814474301" watchObservedRunningTime="2026-07-24 16:21:47.331782706 +0000 UTC m=+2466.816343282"
Jul 24 16:21:49.368486 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:49.368447    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:21:56.386587 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:56.386556    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:56.386995 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:56.386604    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:56.399090 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:56.399068    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:21:57.372294 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:21:57.372261    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:22:28.272564 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.272522    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"]
Jul 24 16:22:28.276023 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.276000    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.280914 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.280897    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"kube-root-ca.crt\""
Jul 24 16:22:28.282059 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.282036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"default-dockercfg-fjkwm\""
Jul 24 16:22:28.282235 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.282120    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"auth-test-model-5aebfb-kserve-self-signed-certs\""
Jul 24 16:22:28.282235 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.282148    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"openshift-service-ca.crt\""
Jul 24 16:22:28.290536 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.290512    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"]
Jul 24 16:22:28.413276 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.413230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-dshm\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.413425 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.413284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tls-certs\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.413425 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.413370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqt4v\" (UniqueName: \"kubernetes.io/projected/cdd855be-18bf-4054-b3e6-5fcedf53ef70-kube-api-access-fqt4v\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.413425 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.413400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tmp-dir\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.413425 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.413422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-home\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.413562 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.413469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-model-cache\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.514808 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.514781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqt4v\" (UniqueName: \"kubernetes.io/projected/cdd855be-18bf-4054-b3e6-5fcedf53ef70-kube-api-access-fqt4v\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.514937 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.514814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tmp-dir\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.514937 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.514840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-home\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.514937 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.514873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-model-cache\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.514937 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.514922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-dshm\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.515147 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.514953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tls-certs\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.515204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.515145    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tmp-dir\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.515279 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.515232    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-home\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.515279 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.515267    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-model-cache\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.517620 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.517601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-dshm\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.517732 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.517700    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tls-certs\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.526507 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.526447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqt4v\" (UniqueName: \"kubernetes.io/projected/cdd855be-18bf-4054-b3e6-5fcedf53ef70-kube-api-access-fqt4v\") pod \"auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") " pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.587087 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.587053    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:28.723858 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:28.723830    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"]
Jul 24 16:22:28.725606 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:22:28.725577    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdd855be_18bf_4054_b3e6_5fcedf53ef70.slice/crio-6cb8722054a7c61314cf2b01084f8283cf6eaef30096128da4e05c07a1564382 WatchSource:0}: Error finding container 6cb8722054a7c61314cf2b01084f8283cf6eaef30096128da4e05c07a1564382: Status 404 returned error can't find the container with id 6cb8722054a7c61314cf2b01084f8283cf6eaef30096128da4e05c07a1564382
Jul 24 16:22:29.529383 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:29.529342    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm" event={"ID":"cdd855be-18bf-4054-b3e6-5fcedf53ef70","Type":"ContainerStarted","Data":"740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c"}
Jul 24 16:22:29.529737 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:29.529391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm" event={"ID":"cdd855be-18bf-4054-b3e6-5fcedf53ef70","Type":"ContainerStarted","Data":"6cb8722054a7c61314cf2b01084f8283cf6eaef30096128da4e05c07a1564382"}
Jul 24 16:22:29.547320 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:29.547273    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm" podStartSLOduration=1.547233666 podStartE2EDuration="1.547233666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:22:29.545975777 +0000 UTC m=+2509.030536376" watchObservedRunningTime="2026-07-24 16:22:29.547233666 +0000 UTC m=+2509.031794263"
Jul 24 16:22:32.067139 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:32.067105    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:22:38.587586 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:38.587550    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:38.587586 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:38.587594    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:38.600300 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:38.600277    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:39.589943 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:39.589912    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:22:50.165384 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:22:50.165351    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:23:07.663562 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:23:07.663521    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:23:24.960883 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:23:24.960847    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:23:42.866605 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:23:42.866567    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:24:00.572934 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:00.572898    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:24:13.066443 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:13.066407    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:24:30.575440 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:30.575403    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:24:43.352696 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:43.352661    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"]
Jul 24 16:24:44.379035 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:44.378999    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"]
Jul 24 16:24:49.234797 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.234764    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:24:49.378050 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.378015    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"]
Jul 24 16:24:49.381297 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.381280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.383896 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.383874    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ed0643\"/\"sub-test-model-ed0643-kserve-self-signed-certs\""
Jul 24 16:24:49.390403 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.390381    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"]
Jul 24 16:24:49.421880 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.421851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-dshm\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.421880 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.421881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-home\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.422046 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.421907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6hk\" (UniqueName: \"kubernetes.io/projected/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-kube-api-access-bm6hk\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.422046 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.421999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tmp-dir\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.422046 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.422029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tls-certs\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.422183 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.422069    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-model-cache\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.523500 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.523411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tmp-dir\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.523500 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.523450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tls-certs\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.523500 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.523480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-model-cache\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.523779 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.523516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-dshm\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.523779 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.523532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-home\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.523779 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.523556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6hk\" (UniqueName: \"kubernetes.io/projected/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-kube-api-access-bm6hk\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.523973 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.523860    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tmp-dir\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.523973 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.523959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-model-cache\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.524061 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.523979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-home\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.525953 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.525931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-dshm\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.526254 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.526218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tls-certs\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.532568 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.532549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6hk\" (UniqueName: \"kubernetes.io/projected/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-kube-api-access-bm6hk\") pod \"sub-test-model-ed0643-kserve-64bc944f87-kwqrm\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") " pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.691675 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.691637    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:49.826840 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:49.826805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"]
Jul 24 16:24:49.829820 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:24:49.829796    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91ef1538_fc34_4e1e_b4e1_2ec3628a073b.slice/crio-24fab8945ecddd6ede6bf23ac2fb7ce6b76b9c97fe135a22023af44abb3d93f8 WatchSource:0}: Error finding container 24fab8945ecddd6ede6bf23ac2fb7ce6b76b9c97fe135a22023af44abb3d93f8: Status 404 returned error can't find the container with id 24fab8945ecddd6ede6bf23ac2fb7ce6b76b9c97fe135a22023af44abb3d93f8
Jul 24 16:24:50.238172 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:50.238134    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm" event={"ID":"91ef1538-fc34-4e1e-b4e1-2ec3628a073b","Type":"ContainerStarted","Data":"857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e"}
Jul 24 16:24:50.238624 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:50.238180    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm" event={"ID":"91ef1538-fc34-4e1e-b4e1-2ec3628a073b","Type":"ContainerStarted","Data":"24fab8945ecddd6ede6bf23ac2fb7ce6b76b9c97fe135a22023af44abb3d93f8"}
Jul 24 16:24:50.257108 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:50.257058    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm" podStartSLOduration=1.257041252 podStartE2EDuration="1.257041252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:24:50.255631102 +0000 UTC m=+2649.740191701" watchObservedRunningTime="2026-07-24 16:24:50.257041252 +0000 UTC m=+2649.741601850"
Jul 24 16:24:58.372669 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.372600    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq" podUID="05005c72-dfa3-499a-af4e-85af8d4d3118" containerName="main" containerID="cri-o://0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7" gracePeriod=45
Jul 24 16:24:58.627534 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.627475    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:24:58.709365 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709333    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-model-cache\") pod \"05005c72-dfa3-499a-af4e-85af8d4d3118\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") "
Jul 24 16:24:58.709529 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709403    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05005c72-dfa3-499a-af4e-85af8d4d3118-tls-certs\") pod \"05005c72-dfa3-499a-af4e-85af8d4d3118\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") "
Jul 24 16:24:58.709529 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709461    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-dshm\") pod \"05005c72-dfa3-499a-af4e-85af8d4d3118\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") "
Jul 24 16:24:58.709529 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709514    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnrfp\" (UniqueName: \"kubernetes.io/projected/05005c72-dfa3-499a-af4e-85af8d4d3118-kube-api-access-tnrfp\") pod \"05005c72-dfa3-499a-af4e-85af8d4d3118\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") "
Jul 24 16:24:58.709688 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709541    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-tmp-dir\") pod \"05005c72-dfa3-499a-af4e-85af8d4d3118\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") "
Jul 24 16:24:58.709688 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709576    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-home\") pod \"05005c72-dfa3-499a-af4e-85af8d4d3118\" (UID: \"05005c72-dfa3-499a-af4e-85af8d4d3118\") "
Jul 24 16:24:58.709688 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709652    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-model-cache" (OuterVolumeSpecName: "model-cache") pod "05005c72-dfa3-499a-af4e-85af8d4d3118" (UID: "05005c72-dfa3-499a-af4e-85af8d4d3118"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:24:58.709847 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709803    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-home" (OuterVolumeSpecName: "home") pod "05005c72-dfa3-499a-af4e-85af8d4d3118" (UID: "05005c72-dfa3-499a-af4e-85af8d4d3118"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:24:58.709911 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709837    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05005c72-dfa3-499a-af4e-85af8d4d3118" (UID: "05005c72-dfa3-499a-af4e-85af8d4d3118"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:24:58.709969 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709951    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:58.710013 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709968    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:58.710013 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.709985    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:58.711960 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.711931    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05005c72-dfa3-499a-af4e-85af8d4d3118-kube-api-access-tnrfp" (OuterVolumeSpecName: "kube-api-access-tnrfp") pod "05005c72-dfa3-499a-af4e-85af8d4d3118" (UID: "05005c72-dfa3-499a-af4e-85af8d4d3118"). InnerVolumeSpecName "kube-api-access-tnrfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:24:58.712073 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.711963    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05005c72-dfa3-499a-af4e-85af8d4d3118-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05005c72-dfa3-499a-af4e-85af8d4d3118" (UID: "05005c72-dfa3-499a-af4e-85af8d4d3118"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:24:58.712073 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.711970    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-dshm" (OuterVolumeSpecName: "dshm") pod "05005c72-dfa3-499a-af4e-85af8d4d3118" (UID: "05005c72-dfa3-499a-af4e-85af8d4d3118"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:24:58.810992 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.810959    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05005c72-dfa3-499a-af4e-85af8d4d3118-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:58.810992 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.810986    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05005c72-dfa3-499a-af4e-85af8d4d3118-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:58.810992 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:58.810995    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrfp\" (UniqueName: \"kubernetes.io/projected/05005c72-dfa3-499a-af4e-85af8d4d3118-kube-api-access-tnrfp\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:59.284349 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.284236    2576 generic.go:356] "Generic (PLEG): container finished" podID="05005c72-dfa3-499a-af4e-85af8d4d3118" containerID="0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7" exitCode=0
Jul 24 16:24:59.284349 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.284269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq" event={"ID":"05005c72-dfa3-499a-af4e-85af8d4d3118","Type":"ContainerDied","Data":"0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7"}
Jul 24 16:24:59.284349 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.284297    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"
Jul 24 16:24:59.284349 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.284323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq" event={"ID":"05005c72-dfa3-499a-af4e-85af8d4d3118","Type":"ContainerDied","Data":"69db1b479ecf051673152653b51100b5b0d99da34f2ec6b13502e3e4d4d4b5d3"}
Jul 24 16:24:59.284349 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.284342    2576 scope.go:117] "RemoveContainer" containerID="0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7"
Jul 24 16:24:59.293208 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.293193    2576 scope.go:117] "RemoveContainer" containerID="0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7"
Jul 24 16:24:59.293504 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:24:59.293488    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7\": container with ID starting with 0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7 not found: ID does not exist" containerID="0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7"
Jul 24 16:24:59.293553 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.293525    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7"} err="failed to get container status \"0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7\": rpc error: code = NotFound desc = could not find container \"0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7\": container with ID starting with 0e824907cabfa5a1e8681575388de8b7c19e06e25c33af480c10ace0fee9b8e7 not found: ID does not exist"
Jul 24 16:24:59.301215 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.301192    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"]
Jul 24 16:24:59.305408 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.305386    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/auth-test-model-ed0643-kserve-8547d85d8c-hvlqq"]
Jul 24 16:24:59.399382 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.399338    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm" podUID="cdd855be-18bf-4054-b3e6-5fcedf53ef70" containerName="main" containerID="cri-o://740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c" gracePeriod=45
Jul 24 16:24:59.660043 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.660018    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:24:59.691775 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.691739    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:59.691957 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.691786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:59.704050 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.704017    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:24:59.717993 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.717966    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-model-cache\") pod \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") "
Jul 24 16:24:59.718151 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718002    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-home\") pod \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") "
Jul 24 16:24:59.718151 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718058    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqt4v\" (UniqueName: \"kubernetes.io/projected/cdd855be-18bf-4054-b3e6-5fcedf53ef70-kube-api-access-fqt4v\") pod \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") "
Jul 24 16:24:59.718151 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718113    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-dshm\") pod \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") "
Jul 24 16:24:59.718151 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718135    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tls-certs\") pod \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") "
Jul 24 16:24:59.718403 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718186    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tmp-dir\") pod \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\" (UID: \"cdd855be-18bf-4054-b3e6-5fcedf53ef70\") "
Jul 24 16:24:59.718403 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718235    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdd855be-18bf-4054-b3e6-5fcedf53ef70" (UID: "cdd855be-18bf-4054-b3e6-5fcedf53ef70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:24:59.718403 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718360    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-home" (OuterVolumeSpecName: "home") pod "cdd855be-18bf-4054-b3e6-5fcedf53ef70" (UID: "cdd855be-18bf-4054-b3e6-5fcedf53ef70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:24:59.718553 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718460    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdd855be-18bf-4054-b3e6-5fcedf53ef70" (UID: "cdd855be-18bf-4054-b3e6-5fcedf53ef70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:24:59.718553 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718509    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:59.718553 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.718528    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:59.720723 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.720696    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdd855be-18bf-4054-b3e6-5fcedf53ef70" (UID: "cdd855be-18bf-4054-b3e6-5fcedf53ef70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:24:59.720865 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.720733    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd855be-18bf-4054-b3e6-5fcedf53ef70-kube-api-access-fqt4v" (OuterVolumeSpecName: "kube-api-access-fqt4v") pod "cdd855be-18bf-4054-b3e6-5fcedf53ef70" (UID: "cdd855be-18bf-4054-b3e6-5fcedf53ef70"). InnerVolumeSpecName "kube-api-access-fqt4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:24:59.721543 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.721516    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-dshm" (OuterVolumeSpecName: "dshm") pod "cdd855be-18bf-4054-b3e6-5fcedf53ef70" (UID: "cdd855be-18bf-4054-b3e6-5fcedf53ef70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:24:59.819046 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.818933    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqt4v\" (UniqueName: \"kubernetes.io/projected/cdd855be-18bf-4054-b3e6-5fcedf53ef70-kube-api-access-fqt4v\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:59.819046 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.818969    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:59.819046 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.818987    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:59.819046 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.819000    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd855be-18bf-4054-b3e6-5fcedf53ef70-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:24:59.967102 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:24:59.967067    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:25:00.291443 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.291415    2576 generic.go:356] "Generic (PLEG): container finished" podID="cdd855be-18bf-4054-b3e6-5fcedf53ef70" containerID="740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c" exitCode=0
Jul 24 16:25:00.291619 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.291457    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"
Jul 24 16:25:00.291619 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.291489    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm" event={"ID":"cdd855be-18bf-4054-b3e6-5fcedf53ef70","Type":"ContainerDied","Data":"740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c"}
Jul 24 16:25:00.291619 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.291517    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm" event={"ID":"cdd855be-18bf-4054-b3e6-5fcedf53ef70","Type":"ContainerDied","Data":"6cb8722054a7c61314cf2b01084f8283cf6eaef30096128da4e05c07a1564382"}
Jul 24 16:25:00.291619 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.291535    2576 scope.go:117] "RemoveContainer" containerID="740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c"
Jul 24 16:25:00.301130 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.301112    2576 scope.go:117] "RemoveContainer" containerID="740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c"
Jul 24 16:25:00.301423 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:25:00.301401    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c\": container with ID starting with 740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c not found: ID does not exist" containerID="740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c"
Jul 24 16:25:00.301509 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.301435    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c"} err="failed to get container status \"740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c\": rpc error: code = NotFound desc = could not find container \"740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c\": container with ID starting with 740d25db2d0c280e91aada42a986c7d8d842d533530bbbd4af5a64e4d9d8262c not found: ID does not exist"
Jul 24 16:25:00.303934 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.303916    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:25:00.313233 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.313210    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"]
Jul 24 16:25:00.319698 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:00.319678    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm"]
Jul 24 16:25:01.005300 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:01.005264    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05005c72-dfa3-499a-af4e-85af8d4d3118" path="/var/lib/kubelet/pods/05005c72-dfa3-499a-af4e-85af8d4d3118/volumes"
Jul 24 16:25:01.005683 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:01.005657    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd855be-18bf-4054-b3e6-5fcedf53ef70" path="/var/lib/kubelet/pods/cdd855be-18bf-4054-b3e6-5fcedf53ef70/volumes"
Jul 24 16:25:04.580104 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:04.580052    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-5aebfb/auth-test-model-5aebfb-kserve-5b7f4d66ff-p9dnm" podUID="cdd855be-18bf-4054-b3e6-5fcedf53ef70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Jul 24 16:25:29.267488 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.267442    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"]
Jul 24 16:25:29.268204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.267914    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05005c72-dfa3-499a-af4e-85af8d4d3118" containerName="main"
Jul 24 16:25:29.268204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.267933    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="05005c72-dfa3-499a-af4e-85af8d4d3118" containerName="main"
Jul 24 16:25:29.268204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.267951    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd855be-18bf-4054-b3e6-5fcedf53ef70" containerName="main"
Jul 24 16:25:29.268204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.267962    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd855be-18bf-4054-b3e6-5fcedf53ef70" containerName="main"
Jul 24 16:25:29.268204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.268066    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd855be-18bf-4054-b3e6-5fcedf53ef70" containerName="main"
Jul 24 16:25:29.268204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.268081    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="05005c72-dfa3-499a-af4e-85af8d4d3118" containerName="main"
Jul 24 16:25:29.271204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.271183    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.274729 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.274707    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"kube-root-ca.crt\""
Jul 24 16:25:29.274847 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.274766    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"openshift-service-ca.crt\""
Jul 24 16:25:29.274847 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.274773    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"sub-test-model-5aebfb-kserve-self-signed-certs\""
Jul 24 16:25:29.274847 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.274801    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"default-dockercfg-fjkwm\""
Jul 24 16:25:29.280297 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.280275    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"]
Jul 24 16:25:29.372378 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.372343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-tmp-dir\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.372559 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.372396    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-home\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.372559 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.372451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba583a29-0878-4138-8500-2132893ec8d6-tls-certs\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.372559 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.372496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-model-cache\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.372559 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.372531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-928jq\" (UniqueName: \"kubernetes.io/projected/ba583a29-0878-4138-8500-2132893ec8d6-kube-api-access-928jq\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.372719 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.372575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-dshm\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.473669 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.473633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-928jq\" (UniqueName: \"kubernetes.io/projected/ba583a29-0878-4138-8500-2132893ec8d6-kube-api-access-928jq\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.473866 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.473690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-dshm\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.473866 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.473716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-tmp-dir\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.473866 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.473748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-home\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.473866 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.473768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba583a29-0878-4138-8500-2132893ec8d6-tls-certs\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.473866 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.473791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-model-cache\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.474168 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.474147    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-home\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.474168 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.474161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-model-cache\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.474311 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.474280    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-tmp-dir\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.476315 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.476289    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-dshm\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.476706 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.476687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba583a29-0878-4138-8500-2132893ec8d6-tls-certs\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.481985 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.481961    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-928jq\" (UniqueName: \"kubernetes.io/projected/ba583a29-0878-4138-8500-2132893ec8d6-kube-api-access-928jq\") pod \"sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") " pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.581214 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.581106    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:29.734402 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:29.734379    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"]
Jul 24 16:25:29.736423 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:25:29.736380    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba583a29_0878_4138_8500_2132893ec8d6.slice/crio-2f660b4472b2502df7b2708301b31c9979921896269d5ffb2a4930c32b0d42d2 WatchSource:0}: Error finding container 2f660b4472b2502df7b2708301b31c9979921896269d5ffb2a4930c32b0d42d2: Status 404 returned error can't find the container with id 2f660b4472b2502df7b2708301b31c9979921896269d5ffb2a4930c32b0d42d2
Jul 24 16:25:30.449410 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:30.449373    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs" event={"ID":"ba583a29-0878-4138-8500-2132893ec8d6","Type":"ContainerStarted","Data":"87007b7cad4f7c05e48e757665566ac22fad2a8ed283837ec65b94e72e36d696"}
Jul 24 16:25:30.449410 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:30.449416    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs" event={"ID":"ba583a29-0878-4138-8500-2132893ec8d6","Type":"ContainerStarted","Data":"2f660b4472b2502df7b2708301b31c9979921896269d5ffb2a4930c32b0d42d2"}
Jul 24 16:25:30.468887 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:30.468834    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs" podStartSLOduration=1.468817496 podStartE2EDuration="1.468817496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:25:30.468085418 +0000 UTC m=+2689.952646017" watchObservedRunningTime="2026-07-24 16:25:30.468817496 +0000 UTC m=+2689.953378093"
Jul 24 16:25:32.969359 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:32.969268    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:25:39.581613 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:39.581579    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:39.581613 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:39.581621    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:39.593579 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:39.593554    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:40.510258 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:40.510224    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:25:41.221030 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:41.220996    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:25:41.225206 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:41.225187    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:25:41.230549 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:41.230530    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:25:41.234791 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:41.234777    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:25:48.070252 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:25:48.070204    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:26:07.179961 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:07.179924    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:26:13.641781 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:13.641746    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"]
Jul 24 16:26:14.556319 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:14.556283    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"]
Jul 24 16:26:25.763659 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:25.763620    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:26:28.661330 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.661266    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm" podUID="91ef1538-fc34-4e1e-b4e1-2ec3628a073b" containerName="main" containerID="cri-o://857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e" gracePeriod=45
Jul 24 16:26:28.913645 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.913591    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:26:28.988222 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.988194    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm6hk\" (UniqueName: \"kubernetes.io/projected/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-kube-api-access-bm6hk\") pod \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") "
Jul 24 16:26:28.988222 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.988226    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tls-certs\") pod \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") "
Jul 24 16:26:28.988480 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.988274    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-dshm\") pod \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") "
Jul 24 16:26:28.988480 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.988326    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tmp-dir\") pod \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") "
Jul 24 16:26:28.988480 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.988365    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-model-cache\") pod \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") "
Jul 24 16:26:28.988480 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.988391    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-home\") pod \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\" (UID: \"91ef1538-fc34-4e1e-b4e1-2ec3628a073b\") "
Jul 24 16:26:28.988688 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.988613    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91ef1538-fc34-4e1e-b4e1-2ec3628a073b" (UID: "91ef1538-fc34-4e1e-b4e1-2ec3628a073b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:26:28.988688 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.988663    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-model-cache" (OuterVolumeSpecName: "model-cache") pod "91ef1538-fc34-4e1e-b4e1-2ec3628a073b" (UID: "91ef1538-fc34-4e1e-b4e1-2ec3628a073b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:26:28.988782 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.988758    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-home" (OuterVolumeSpecName: "home") pod "91ef1538-fc34-4e1e-b4e1-2ec3628a073b" (UID: "91ef1538-fc34-4e1e-b4e1-2ec3628a073b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:26:28.990615 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.990590    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-kube-api-access-bm6hk" (OuterVolumeSpecName: "kube-api-access-bm6hk") pod "91ef1538-fc34-4e1e-b4e1-2ec3628a073b" (UID: "91ef1538-fc34-4e1e-b4e1-2ec3628a073b"). InnerVolumeSpecName "kube-api-access-bm6hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:26:28.990732 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.990638    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-dshm" (OuterVolumeSpecName: "dshm") pod "91ef1538-fc34-4e1e-b4e1-2ec3628a073b" (UID: "91ef1538-fc34-4e1e-b4e1-2ec3628a073b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:26:28.990732 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:28.990636    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91ef1538-fc34-4e1e-b4e1-2ec3628a073b" (UID: "91ef1538-fc34-4e1e-b4e1-2ec3628a073b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:26:29.089551 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.089522    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.089551 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.089545    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.089551 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.089553    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm6hk\" (UniqueName: \"kubernetes.io/projected/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-kube-api-access-bm6hk\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.089551 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.089562    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.089814 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.089570    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.089814 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.089578    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ef1538-fc34-4e1e-b4e1-2ec3628a073b-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.575742 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.575678    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs" podUID="ba583a29-0878-4138-8500-2132893ec8d6" containerName="main" containerID="cri-o://87007b7cad4f7c05e48e757665566ac22fad2a8ed283837ec65b94e72e36d696" gracePeriod=45
Jul 24 16:26:29.756838 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.756808    2576 generic.go:356] "Generic (PLEG): container finished" podID="ba583a29-0878-4138-8500-2132893ec8d6" containerID="87007b7cad4f7c05e48e757665566ac22fad2a8ed283837ec65b94e72e36d696" exitCode=0
Jul 24 16:26:29.757302 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.756879    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs" event={"ID":"ba583a29-0878-4138-8500-2132893ec8d6","Type":"ContainerDied","Data":"87007b7cad4f7c05e48e757665566ac22fad2a8ed283837ec65b94e72e36d696"}
Jul 24 16:26:29.758341 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.758317    2576 generic.go:356] "Generic (PLEG): container finished" podID="91ef1538-fc34-4e1e-b4e1-2ec3628a073b" containerID="857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e" exitCode=0
Jul 24 16:26:29.758465 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.758369    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"
Jul 24 16:26:29.758530 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.758340    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm" event={"ID":"91ef1538-fc34-4e1e-b4e1-2ec3628a073b","Type":"ContainerDied","Data":"857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e"}
Jul 24 16:26:29.758530 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.758509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm" event={"ID":"91ef1538-fc34-4e1e-b4e1-2ec3628a073b","Type":"ContainerDied","Data":"24fab8945ecddd6ede6bf23ac2fb7ce6b76b9c97fe135a22023af44abb3d93f8"}
Jul 24 16:26:29.758651 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.758538    2576 scope.go:117] "RemoveContainer" containerID="857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e"
Jul 24 16:26:29.769903 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.769874    2576 scope.go:117] "RemoveContainer" containerID="857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e"
Jul 24 16:26:29.770178 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:26:29.770161    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e\": container with ID starting with 857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e not found: ID does not exist" containerID="857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e"
Jul 24 16:26:29.770252 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.770187    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e"} err="failed to get container status \"857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e\": rpc error: code = NotFound desc = could not find container \"857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e\": container with ID starting with 857ab9ffa589b4d8fc1671d74a555c6b7d88b15378d9de63462eeb2b01b4486e not found: ID does not exist"
Jul 24 16:26:29.776890 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.776849    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"]
Jul 24 16:26:29.781126 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.781102    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/sub-test-model-ed0643-kserve-64bc944f87-kwqrm"]
Jul 24 16:26:29.830684 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.830633    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:26:29.896535 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.896510    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba583a29-0878-4138-8500-2132893ec8d6-tls-certs\") pod \"ba583a29-0878-4138-8500-2132893ec8d6\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") "
Jul 24 16:26:29.896722 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.896548    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-dshm\") pod \"ba583a29-0878-4138-8500-2132893ec8d6\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") "
Jul 24 16:26:29.896722 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.896575    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-model-cache\") pod \"ba583a29-0878-4138-8500-2132893ec8d6\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") "
Jul 24 16:26:29.896722 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.896591    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-928jq\" (UniqueName: \"kubernetes.io/projected/ba583a29-0878-4138-8500-2132893ec8d6-kube-api-access-928jq\") pod \"ba583a29-0878-4138-8500-2132893ec8d6\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") "
Jul 24 16:26:29.896722 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.896616    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-home\") pod \"ba583a29-0878-4138-8500-2132893ec8d6\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") "
Jul 24 16:26:29.896722 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.896709    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-tmp-dir\") pod \"ba583a29-0878-4138-8500-2132893ec8d6\" (UID: \"ba583a29-0878-4138-8500-2132893ec8d6\") "
Jul 24 16:26:29.896984 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.896806    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba583a29-0878-4138-8500-2132893ec8d6" (UID: "ba583a29-0878-4138-8500-2132893ec8d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:26:29.896984 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.896915    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-home" (OuterVolumeSpecName: "home") pod "ba583a29-0878-4138-8500-2132893ec8d6" (UID: "ba583a29-0878-4138-8500-2132893ec8d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:26:29.896984 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.896947    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba583a29-0878-4138-8500-2132893ec8d6" (UID: "ba583a29-0878-4138-8500-2132893ec8d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:26:29.897097 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.897004    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.897097 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.897023    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.897097 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.897035    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.898805 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.898776    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-dshm" (OuterVolumeSpecName: "dshm") pod "ba583a29-0878-4138-8500-2132893ec8d6" (UID: "ba583a29-0878-4138-8500-2132893ec8d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:26:29.898908 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.898842    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba583a29-0878-4138-8500-2132893ec8d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba583a29-0878-4138-8500-2132893ec8d6" (UID: "ba583a29-0878-4138-8500-2132893ec8d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:26:29.898976 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.898952    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba583a29-0878-4138-8500-2132893ec8d6-kube-api-access-928jq" (OuterVolumeSpecName: "kube-api-access-928jq") pod "ba583a29-0878-4138-8500-2132893ec8d6" (UID: "ba583a29-0878-4138-8500-2132893ec8d6"). InnerVolumeSpecName "kube-api-access-928jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:26:29.997437 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.997406    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba583a29-0878-4138-8500-2132893ec8d6-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.997437 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.997432    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba583a29-0878-4138-8500-2132893ec8d6-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:29.997618 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:29.997444    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-928jq\" (UniqueName: \"kubernetes.io/projected/ba583a29-0878-4138-8500-2132893ec8d6-kube-api-access-928jq\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:26:30.764914 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:30.764873    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"
Jul 24 16:26:30.765427 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:30.764915    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs" event={"ID":"ba583a29-0878-4138-8500-2132893ec8d6","Type":"ContainerDied","Data":"2f660b4472b2502df7b2708301b31c9979921896269d5ffb2a4930c32b0d42d2"}
Jul 24 16:26:30.765427 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:30.764970    2576 scope.go:117] "RemoveContainer" containerID="87007b7cad4f7c05e48e757665566ac22fad2a8ed283837ec65b94e72e36d696"
Jul 24 16:26:30.787478 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:30.787439    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"]
Jul 24 16:26:30.791360 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:30.791338    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/sub-test-model-5aebfb-kserve-6f88d49bcc-jqrqs"]
Jul 24 16:26:31.005531 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.005494    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ef1538-fc34-4e1e-b4e1-2ec3628a073b" path="/var/lib/kubelet/pods/91ef1538-fc34-4e1e-b4e1-2ec3628a073b/volumes"
Jul 24 16:26:31.005871 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.005858    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba583a29-0878-4138-8500-2132893ec8d6" path="/var/lib/kubelet/pods/ba583a29-0878-4138-8500-2132893ec8d6/volumes"
Jul 24 16:26:31.952664 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.952630    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"]
Jul 24 16:26:31.953031 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.952982    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ef1538-fc34-4e1e-b4e1-2ec3628a073b" containerName="main"
Jul 24 16:26:31.953031 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.952993    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ef1538-fc34-4e1e-b4e1-2ec3628a073b" containerName="main"
Jul 24 16:26:31.953031 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.953004    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba583a29-0878-4138-8500-2132893ec8d6" containerName="main"
Jul 24 16:26:31.953031 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.953009    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba583a29-0878-4138-8500-2132893ec8d6" containerName="main"
Jul 24 16:26:31.953164 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.953073    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba583a29-0878-4138-8500-2132893ec8d6" containerName="main"
Jul 24 16:26:31.953164 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.953082    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ef1538-fc34-4e1e-b4e1-2ec3628a073b" containerName="main"
Jul 24 16:26:31.957131 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.957113    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:31.960970 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.960943    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ed0643\"/\"rate-test-model-ed0643-kserve-self-signed-certs\""
Jul 24 16:26:31.960970 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.960954    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ed0643\"/\"default-dockercfg-n6jxn\""
Jul 24 16:26:31.961207 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.961188    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ed0643\"/\"kube-root-ca.crt\""
Jul 24 16:26:31.961497 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.961481    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ed0643\"/\"openshift-service-ca.crt\""
Jul 24 16:26:31.969182 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:31.969159    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"]
Jul 24 16:26:32.012380 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.012348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042fdd01-92eb-4004-8333-6baf81e1d2ea-tls-certs\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.012635 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.012404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-dshm\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.012635 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.012425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65r9r\" (UniqueName: \"kubernetes.io/projected/042fdd01-92eb-4004-8333-6baf81e1d2ea-kube-api-access-65r9r\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.012635 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.012505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-home\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.012635 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.012534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-model-cache\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.012635 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.012555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-tmp-dir\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.113437 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.113403    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-dshm\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.113437 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.113441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r9r\" (UniqueName: \"kubernetes.io/projected/042fdd01-92eb-4004-8333-6baf81e1d2ea-kube-api-access-65r9r\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.113618 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.113576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-home\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.113618 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.113611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-model-cache\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.113746 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.113640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-tmp-dir\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.113746 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.113711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042fdd01-92eb-4004-8333-6baf81e1d2ea-tls-certs\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.114007 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.113986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-home\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.114075 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.114009    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-model-cache\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.114075 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.114036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-tmp-dir\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.115880 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.115863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-dshm\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.116217 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.116201    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042fdd01-92eb-4004-8333-6baf81e1d2ea-tls-certs\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.121627 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.121601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65r9r\" (UniqueName: \"kubernetes.io/projected/042fdd01-92eb-4004-8333-6baf81e1d2ea-kube-api-access-65r9r\") pod \"rate-test-model-ed0643-kserve-575d6c8b4c-d54lx\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") " pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.266727 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.266628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:32.405052 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.405028    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"]
Jul 24 16:26:32.407024 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:26:32.406996    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod042fdd01_92eb_4004_8333_6baf81e1d2ea.slice/crio-78c01177a308b3839e6d709cfe70a8d63a6030f15fcbdded102a8bacdb1d5de6 WatchSource:0}: Error finding container 78c01177a308b3839e6d709cfe70a8d63a6030f15fcbdded102a8bacdb1d5de6: Status 404 returned error can't find the container with id 78c01177a308b3839e6d709cfe70a8d63a6030f15fcbdded102a8bacdb1d5de6
Jul 24 16:26:32.778590 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.778555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx" event={"ID":"042fdd01-92eb-4004-8333-6baf81e1d2ea","Type":"ContainerStarted","Data":"f66586ac9f808e07c390bb6b5a15eff9d5e07721bef013c22f47d3e25d694aeb"}
Jul 24 16:26:32.778765 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.778599    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx" event={"ID":"042fdd01-92eb-4004-8333-6baf81e1d2ea","Type":"ContainerStarted","Data":"78c01177a308b3839e6d709cfe70a8d63a6030f15fcbdded102a8bacdb1d5de6"}
Jul 24 16:26:32.797583 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:32.797538    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx" podStartSLOduration=1.797524473 podStartE2EDuration="1.797524473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:26:32.795832843 +0000 UTC m=+2752.280393444" watchObservedRunningTime="2026-07-24 16:26:32.797524473 +0000 UTC m=+2752.282085070"
Jul 24 16:26:34.486681 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:34.486645    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:26:42.267382 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:42.267351    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:42.267776 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:42.267407    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:42.279717 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:42.279693    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:26:42.837671 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:26:42.837644    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:27:11.272549 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.272506    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"]
Jul 24 16:27:11.276198 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.276176    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.278938 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.278912    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"kube-root-ca.crt\""
Jul 24 16:27:11.279067 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.279014    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"openshift-service-ca.crt\""
Jul 24 16:27:11.280450 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.280426    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"default-dockercfg-fjkwm\""
Jul 24 16:27:11.280579 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.280456    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5aebfb\"/\"rate-test-model-5aebfb-kserve-self-signed-certs\""
Jul 24 16:27:11.285668 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.285645    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"]
Jul 24 16:27:11.446491 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.446456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tls-certs\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.446491 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.446490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-home\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.446748 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.446510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-model-cache\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.446748 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.446583    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tmp-dir\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.446748 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.446668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-dshm\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.446748 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.446693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7tkm\" (UniqueName: \"kubernetes.io/projected/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-kube-api-access-k7tkm\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.548034 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.547953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tls-certs\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.548034 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.547994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-home\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.548034 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.548012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-model-cache\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.548330 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.548043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tmp-dir\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.548330 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.548128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-dshm\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.548330 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.548157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7tkm\" (UniqueName: \"kubernetes.io/projected/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-kube-api-access-k7tkm\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.548502 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.548400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-home\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.548502 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.548455    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-model-cache\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.548502 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.548495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tmp-dir\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.550440 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.550423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-dshm\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.550529 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.550514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tls-certs\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.557537 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.557517    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7tkm\" (UniqueName: \"kubernetes.io/projected/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-kube-api-access-k7tkm\") pod \"rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") " pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.586501 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.586479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:11.721580 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.721541    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"]
Jul 24 16:27:11.722926 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:27:11.722893    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2b868d4_79c5_4e1f_b753_3c31ea907cb1.slice/crio-77251ecfc8f5f723fc3262e7dc7963bce055201fe446bb3656b1ae745df5e97f WatchSource:0}: Error finding container 77251ecfc8f5f723fc3262e7dc7963bce055201fe446bb3656b1ae745df5e97f: Status 404 returned error can't find the container with id 77251ecfc8f5f723fc3262e7dc7963bce055201fe446bb3656b1ae745df5e97f
Jul 24 16:27:11.969124 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.969086    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq" event={"ID":"d2b868d4-79c5-4e1f-b753-3c31ea907cb1","Type":"ContainerStarted","Data":"013cda53a57900000ffb05482fee4ad5c2641da5cc82d2f4698611a7dd0bdc49"}
Jul 24 16:27:11.969124 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.969134    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq" event={"ID":"d2b868d4-79c5-4e1f-b753-3c31ea907cb1","Type":"ContainerStarted","Data":"77251ecfc8f5f723fc3262e7dc7963bce055201fe446bb3656b1ae745df5e97f"}
Jul 24 16:27:11.987687 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:11.987643    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq" podStartSLOduration=0.987629308 podStartE2EDuration="987.629308ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:27:11.985878836 +0000 UTC m=+2791.470439434" watchObservedRunningTime="2026-07-24 16:27:11.987629308 +0000 UTC m=+2791.472189904"
Jul 24 16:27:14.768666 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:14.768629    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:27:21.586839 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:21.586807    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:21.587223 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:21.586856    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:21.599117 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:21.599094    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:22.031228 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:22.031199    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:27:26.764079 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:26.764043    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:27:43.961191 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:43.961152    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:27:48.089683 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:48.089648    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"]
Jul 24 16:27:49.050532 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:49.050497    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"]
Jul 24 16:27:50.012929 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.012897    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7560-140479636793904"]
Jul 24 16:27:50.016129 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.016114    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479636793904"
Jul 24 16:27:50.018852 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.018828    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p64dw\""
Jul 24 16:27:50.021474 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.021449    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479636793904"]
Jul 24 16:27:50.082670 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.082614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6sl8\" (UniqueName: \"kubernetes.io/projected/7536274b-13f3-4d49-aa08-b96f669784ee-kube-api-access-v6sl8\") pod \"test-curl-7560-140479636793904\" (UID: \"7536274b-13f3-4d49-aa08-b96f669784ee\") " pod="opendatahub/test-curl-7560-140479636793904"
Jul 24 16:27:50.183634 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.183595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6sl8\" (UniqueName: \"kubernetes.io/projected/7536274b-13f3-4d49-aa08-b96f669784ee-kube-api-access-v6sl8\") pod \"test-curl-7560-140479636793904\" (UID: \"7536274b-13f3-4d49-aa08-b96f669784ee\") " pod="opendatahub/test-curl-7560-140479636793904"
Jul 24 16:27:50.191785 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.191759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6sl8\" (UniqueName: \"kubernetes.io/projected/7536274b-13f3-4d49-aa08-b96f669784ee-kube-api-access-v6sl8\") pod \"test-curl-7560-140479636793904\" (UID: \"7536274b-13f3-4d49-aa08-b96f669784ee\") " pod="opendatahub/test-curl-7560-140479636793904"
Jul 24 16:27:50.325869 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.325782    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479636793904"
Jul 24 16:27:50.462287 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.459310    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479636793904"]
Jul 24 16:27:50.466297 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:27:50.466268    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7536274b_13f3_4d49_aa08_b96f669784ee.slice/crio-fff5713a9fa231128a2406e93f014548060a9319d2ee52b604593fa556337689 WatchSource:0}: Error finding container fff5713a9fa231128a2406e93f014548060a9319d2ee52b604593fa556337689: Status 404 returned error can't find the container with id fff5713a9fa231128a2406e93f014548060a9319d2ee52b604593fa556337689
Jul 24 16:27:50.467594 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:50.467573    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 16:27:51.168997 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:51.168965    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479636793904" event={"ID":"7536274b-13f3-4d49-aa08-b96f669784ee","Type":"ContainerStarted","Data":"fff5713a9fa231128a2406e93f014548060a9319d2ee52b604593fa556337689"}
Jul 24 16:27:52.175424 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:52.175393    2576 generic.go:356] "Generic (PLEG): container finished" podID="7536274b-13f3-4d49-aa08-b96f669784ee" containerID="4f9f558e653fee103290d9b3a623c0ba6a7db8b7b7b885e98e62e46645ab5bb6" exitCode=0
Jul 24 16:27:52.175826 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:52.175483    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479636793904" event={"ID":"7536274b-13f3-4d49-aa08-b96f669784ee","Type":"ContainerDied","Data":"4f9f558e653fee103290d9b3a623c0ba6a7db8b7b7b885e98e62e46645ab5bb6"}
Jul 24 16:27:53.314343 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.314317    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479636793904"
Jul 24 16:27:53.350625 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.350595    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7560-140479636793904_7536274b-13f3-4d49-aa08-b96f669784ee/test-curl-7560-140479636793904/0.log"
Jul 24 16:27:53.374513 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.374483    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7560-140479636793904"]
Jul 24 16:27:53.376476 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.376458    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7560-140479636793904"]
Jul 24 16:27:53.416143 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.416115    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6sl8\" (UniqueName: \"kubernetes.io/projected/7536274b-13f3-4d49-aa08-b96f669784ee-kube-api-access-v6sl8\") pod \"7536274b-13f3-4d49-aa08-b96f669784ee\" (UID: \"7536274b-13f3-4d49-aa08-b96f669784ee\") "
Jul 24 16:27:53.418499 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.418478    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7536274b-13f3-4d49-aa08-b96f669784ee-kube-api-access-v6sl8" (OuterVolumeSpecName: "kube-api-access-v6sl8") pod "7536274b-13f3-4d49-aa08-b96f669784ee" (UID: "7536274b-13f3-4d49-aa08-b96f669784ee"). InnerVolumeSpecName "kube-api-access-v6sl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:27:53.497790 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.497708    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7560-140479636793520"]
Jul 24 16:27:53.498060 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.498048    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7536274b-13f3-4d49-aa08-b96f669784ee" containerName="test-curl-7560-140479636793904"
Jul 24 16:27:53.498104 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.498062    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7536274b-13f3-4d49-aa08-b96f669784ee" containerName="test-curl-7560-140479636793904"
Jul 24 16:27:53.498139 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.498130    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7536274b-13f3-4d49-aa08-b96f669784ee" containerName="test-curl-7560-140479636793904"
Jul 24 16:27:53.501066 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.501046    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479636793520"
Jul 24 16:27:53.507717 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.507688    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479636793520"]
Jul 24 16:27:53.516871 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.516845    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6sl8\" (UniqueName: \"kubernetes.io/projected/7536274b-13f3-4d49-aa08-b96f669784ee-kube-api-access-v6sl8\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:27:53.617875 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.617838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fnmh\" (UniqueName: \"kubernetes.io/projected/cf7310af-dbaa-4c0a-9ffb-7e806f274901-kube-api-access-6fnmh\") pod \"test-curl-7560-140479636793520\" (UID: \"cf7310af-dbaa-4c0a-9ffb-7e806f274901\") " pod="opendatahub/test-curl-7560-140479636793520"
Jul 24 16:27:53.718360 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.718329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fnmh\" (UniqueName: \"kubernetes.io/projected/cf7310af-dbaa-4c0a-9ffb-7e806f274901-kube-api-access-6fnmh\") pod \"test-curl-7560-140479636793520\" (UID: \"cf7310af-dbaa-4c0a-9ffb-7e806f274901\") " pod="opendatahub/test-curl-7560-140479636793520"
Jul 24 16:27:53.726700 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.726669    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fnmh\" (UniqueName: \"kubernetes.io/projected/cf7310af-dbaa-4c0a-9ffb-7e806f274901-kube-api-access-6fnmh\") pod \"test-curl-7560-140479636793520\" (UID: \"cf7310af-dbaa-4c0a-9ffb-7e806f274901\") " pod="opendatahub/test-curl-7560-140479636793520"
Jul 24 16:27:53.811716 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.811625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479636793520"
Jul 24 16:27:53.941528 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:53.941500    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479636793520"]
Jul 24 16:27:53.943326 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:27:53.943297    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf7310af_dbaa_4c0a_9ffb_7e806f274901.slice/crio-402ff0c3bfe33c0bcccc71d10a18fa4acef1e849c0af43cdedc758cf6551962a WatchSource:0}: Error finding container 402ff0c3bfe33c0bcccc71d10a18fa4acef1e849c0af43cdedc758cf6551962a: Status 404 returned error can't find the container with id 402ff0c3bfe33c0bcccc71d10a18fa4acef1e849c0af43cdedc758cf6551962a
Jul 24 16:27:54.187409 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:54.187374    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479636793520" event={"ID":"cf7310af-dbaa-4c0a-9ffb-7e806f274901","Type":"ContainerStarted","Data":"402ff0c3bfe33c0bcccc71d10a18fa4acef1e849c0af43cdedc758cf6551962a"}
Jul 24 16:27:54.188523 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:54.188502    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479636793904"
Jul 24 16:27:54.188637 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:54.188510    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fff5713a9fa231128a2406e93f014548060a9319d2ee52b604593fa556337689"
Jul 24 16:27:54.199029 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:27:54.199001    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7560-140479636793904\" is forbidden: User \"system:node:ip-10-0-131-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-210.ec2.internal' and this object" podUID="7536274b-13f3-4d49-aa08-b96f669784ee" pod="opendatahub/test-curl-7560-140479636793904"
Jul 24 16:27:55.005172 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:55.005137    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7536274b-13f3-4d49-aa08-b96f669784ee" path="/var/lib/kubelet/pods/7536274b-13f3-4d49-aa08-b96f669784ee/volumes"
Jul 24 16:27:55.194926 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:55.194890    2576 generic.go:356] "Generic (PLEG): container finished" podID="cf7310af-dbaa-4c0a-9ffb-7e806f274901" containerID="e2c59c290f1441a9738ecba73cb49cc2d3e5204618d9d8bf04061577434cebfd" exitCode=0
Jul 24 16:27:55.195098 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:55.194950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479636793520" event={"ID":"cf7310af-dbaa-4c0a-9ffb-7e806f274901","Type":"ContainerDied","Data":"e2c59c290f1441a9738ecba73cb49cc2d3e5204618d9d8bf04061577434cebfd"}
Jul 24 16:27:56.335120 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.335099    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479636793520"
Jul 24 16:27:56.354311 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.354286    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7560-140479636793520_cf7310af-dbaa-4c0a-9ffb-7e806f274901/test-curl-7560-140479636793520/0.log"
Jul 24 16:27:56.379363 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.379336    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7560-140479636793520"]
Jul 24 16:27:56.383724 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.383697    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7560-140479636793520"]
Jul 24 16:27:56.439326 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.439301    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fnmh\" (UniqueName: \"kubernetes.io/projected/cf7310af-dbaa-4c0a-9ffb-7e806f274901-kube-api-access-6fnmh\") pod \"cf7310af-dbaa-4c0a-9ffb-7e806f274901\" (UID: \"cf7310af-dbaa-4c0a-9ffb-7e806f274901\") "
Jul 24 16:27:56.441464 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.441444    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7310af-dbaa-4c0a-9ffb-7e806f274901-kube-api-access-6fnmh" (OuterVolumeSpecName: "kube-api-access-6fnmh") pod "cf7310af-dbaa-4c0a-9ffb-7e806f274901" (UID: "cf7310af-dbaa-4c0a-9ffb-7e806f274901"). InnerVolumeSpecName "kube-api-access-6fnmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:27:56.540075 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.540002    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fnmh\" (UniqueName: \"kubernetes.io/projected/cf7310af-dbaa-4c0a-9ffb-7e806f274901-kube-api-access-6fnmh\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:27:56.551791 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.551763    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:27:56.552125 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.552113    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7310af-dbaa-4c0a-9ffb-7e806f274901" containerName="test-curl-7560-140479636793520"
Jul 24 16:27:56.552168 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.552126    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7310af-dbaa-4c0a-9ffb-7e806f274901" containerName="test-curl-7560-140479636793520"
Jul 24 16:27:56.552202 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.552181    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7310af-dbaa-4c0a-9ffb-7e806f274901" containerName="test-curl-7560-140479636793520"
Jul 24 16:27:56.555033 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.555019    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:56.559456 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.559431    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:27:56.640402 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.640367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9kf4\" (UniqueName: \"kubernetes.io/projected/7626822e-a689-4c62-8779-0eaf11daedfa-kube-api-access-j9kf4\") pod \"test-curl-7560-140479637038688\" (UID: \"7626822e-a689-4c62-8779-0eaf11daedfa\") " pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:56.741234 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.741201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9kf4\" (UniqueName: \"kubernetes.io/projected/7626822e-a689-4c62-8779-0eaf11daedfa-kube-api-access-j9kf4\") pod \"test-curl-7560-140479637038688\" (UID: \"7626822e-a689-4c62-8779-0eaf11daedfa\") " pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:56.750023 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.749996    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9kf4\" (UniqueName: \"kubernetes.io/projected/7626822e-a689-4c62-8779-0eaf11daedfa-kube-api-access-j9kf4\") pod \"test-curl-7560-140479637038688\" (UID: \"7626822e-a689-4c62-8779-0eaf11daedfa\") " pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:56.864867 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.864793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:56.995951 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:56.995925    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:27:56.997460 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:27:56.997439    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7626822e_a689_4c62_8779_0eaf11daedfa.slice/crio-52a8c17889095a1943a95041619b5bb7c9d063f89e534001fc341fb1302b25f7 WatchSource:0}: Error finding container 52a8c17889095a1943a95041619b5bb7c9d063f89e534001fc341fb1302b25f7: Status 404 returned error can't find the container with id 52a8c17889095a1943a95041619b5bb7c9d063f89e534001fc341fb1302b25f7
Jul 24 16:27:57.005347 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:57.005324    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf7310af-dbaa-4c0a-9ffb-7e806f274901" path="/var/lib/kubelet/pods/cf7310af-dbaa-4c0a-9ffb-7e806f274901/volumes"
Jul 24 16:27:57.207084 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:57.207055    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479636793520"
Jul 24 16:27:57.207084 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:57.207072    2576 scope.go:117] "RemoveContainer" containerID="e2c59c290f1441a9738ecba73cb49cc2d3e5204618d9d8bf04061577434cebfd"
Jul 24 16:27:57.208468 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:57.208403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479637038688" event={"ID":"7626822e-a689-4c62-8779-0eaf11daedfa","Type":"ContainerStarted","Data":"52a8c17889095a1943a95041619b5bb7c9d063f89e534001fc341fb1302b25f7"}
Jul 24 16:27:58.215801 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:58.215771    2576 generic.go:356] "Generic (PLEG): container finished" podID="7626822e-a689-4c62-8779-0eaf11daedfa" containerID="3463be2db871b93406d71230214a56f9a9c08291c60f22139bda5ed9a6b1b4f0" exitCode=0
Jul 24 16:27:58.216181 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:58.215809    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479637038688" event={"ID":"7626822e-a689-4c62-8779-0eaf11daedfa","Type":"ContainerDied","Data":"3463be2db871b93406d71230214a56f9a9c08291c60f22139bda5ed9a6b1b4f0"}
Jul 24 16:27:59.358652 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.358627    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:59.382217 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.382188    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7560-140479637038688_7626822e-a689-4c62-8779-0eaf11daedfa/test-curl-7560-140479637038688/0.log"
Jul 24 16:27:59.409815 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.409785    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:27:59.412622 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.412597    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:27:59.466294 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.466269    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9kf4\" (UniqueName: \"kubernetes.io/projected/7626822e-a689-4c62-8779-0eaf11daedfa-kube-api-access-j9kf4\") pod \"7626822e-a689-4c62-8779-0eaf11daedfa\" (UID: \"7626822e-a689-4c62-8779-0eaf11daedfa\") "
Jul 24 16:27:59.468468 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.468440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7626822e-a689-4c62-8779-0eaf11daedfa-kube-api-access-j9kf4" (OuterVolumeSpecName: "kube-api-access-j9kf4") pod "7626822e-a689-4c62-8779-0eaf11daedfa" (UID: "7626822e-a689-4c62-8779-0eaf11daedfa"). InnerVolumeSpecName "kube-api-access-j9kf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:27:59.533980 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.533945    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:27:59.534321 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.534308    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7626822e-a689-4c62-8779-0eaf11daedfa" containerName="test-curl-7560-140479637038688"
Jul 24 16:27:59.534373 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.534324    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7626822e-a689-4c62-8779-0eaf11daedfa" containerName="test-curl-7560-140479637038688"
Jul 24 16:27:59.534408 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.534382    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7626822e-a689-4c62-8779-0eaf11daedfa" containerName="test-curl-7560-140479637038688"
Jul 24 16:27:59.537409 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.537394    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:59.543602 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.543579    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:27:59.567381 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.567351    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9kf4\" (UniqueName: \"kubernetes.io/projected/7626822e-a689-4c62-8779-0eaf11daedfa-kube-api-access-j9kf4\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:27:59.667818 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.667788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5wff\" (UniqueName: \"kubernetes.io/projected/35e79149-3d3f-4037-acc4-047054a5f1eb-kube-api-access-t5wff\") pod \"test-curl-7560-140479637038688\" (UID: \"35e79149-3d3f-4037-acc4-047054a5f1eb\") " pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:59.769199 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.769117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5wff\" (UniqueName: \"kubernetes.io/projected/35e79149-3d3f-4037-acc4-047054a5f1eb-kube-api-access-t5wff\") pod \"test-curl-7560-140479637038688\" (UID: \"35e79149-3d3f-4037-acc4-047054a5f1eb\") " pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:59.777374 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.777351    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5wff\" (UniqueName: \"kubernetes.io/projected/35e79149-3d3f-4037-acc4-047054a5f1eb-kube-api-access-t5wff\") pod \"test-curl-7560-140479637038688\" (UID: \"35e79149-3d3f-4037-acc4-047054a5f1eb\") " pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:59.847664 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.847635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:27:59.977849 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:27:59.977820    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:27:59.979495 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:27:59.979467    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35e79149_3d3f_4037_acc4_047054a5f1eb.slice/crio-2d0a9c4587e66c9ce0545a2bc8b087ed8beb5af4b146e45f980c6c8895ab2e1d WatchSource:0}: Error finding container 2d0a9c4587e66c9ce0545a2bc8b087ed8beb5af4b146e45f980c6c8895ab2e1d: Status 404 returned error can't find the container with id 2d0a9c4587e66c9ce0545a2bc8b087ed8beb5af4b146e45f980c6c8895ab2e1d
Jul 24 16:28:00.229427 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:00.229401    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a8c17889095a1943a95041619b5bb7c9d063f89e534001fc341fb1302b25f7"
Jul 24 16:28:00.229427 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:00.229405    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:28:00.230655 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:00.230626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479637038688" event={"ID":"35e79149-3d3f-4037-acc4-047054a5f1eb","Type":"ContainerStarted","Data":"2d0a9c4587e66c9ce0545a2bc8b087ed8beb5af4b146e45f980c6c8895ab2e1d"}
Jul 24 16:28:00.241008 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:00.240981    2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7560-140479637038688" oldPodUID="7626822e-a689-4c62-8779-0eaf11daedfa" podUID="35e79149-3d3f-4037-acc4-047054a5f1eb"
Jul 24 16:28:00.369728 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:00.369697    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:28:01.006004 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:01.005972    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7626822e-a689-4c62-8779-0eaf11daedfa" path="/var/lib/kubelet/pods/7626822e-a689-4c62-8779-0eaf11daedfa/volumes"
Jul 24 16:28:01.237737 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:01.237707    2576 generic.go:356] "Generic (PLEG): container finished" podID="35e79149-3d3f-4037-acc4-047054a5f1eb" containerID="1d8034e39f6687e9d19496e69c9ac4e63b479f9af4ad8427a0d8f06cfd4f124b" exitCode=0
Jul 24 16:28:01.237916 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:01.237791    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479637038688" event={"ID":"35e79149-3d3f-4037-acc4-047054a5f1eb","Type":"ContainerDied","Data":"1d8034e39f6687e9d19496e69c9ac4e63b479f9af4ad8427a0d8f06cfd4f124b"}
Jul 24 16:28:02.373924 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.373895    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:28:02.393016 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.392990    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7560-140479637038688_35e79149-3d3f-4037-acc4-047054a5f1eb/test-curl-7560-140479637038688/0.log"
Jul 24 16:28:02.418345 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.418315    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:28:02.420557 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.420530    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7560-140479637038688"]
Jul 24 16:28:02.495709 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.495623    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5wff\" (UniqueName: \"kubernetes.io/projected/35e79149-3d3f-4037-acc4-047054a5f1eb-kube-api-access-t5wff\") pod \"35e79149-3d3f-4037-acc4-047054a5f1eb\" (UID: \"35e79149-3d3f-4037-acc4-047054a5f1eb\") "
Jul 24 16:28:02.497845 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.497817    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e79149-3d3f-4037-acc4-047054a5f1eb-kube-api-access-t5wff" (OuterVolumeSpecName: "kube-api-access-t5wff") pod "35e79149-3d3f-4037-acc4-047054a5f1eb" (UID: "35e79149-3d3f-4037-acc4-047054a5f1eb"). InnerVolumeSpecName "kube-api-access-t5wff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:28:02.534163 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.534135    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:02.534483 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.534470    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e79149-3d3f-4037-acc4-047054a5f1eb" containerName="test-curl-7560-140479637038688"
Jul 24 16:28:02.534536 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.534484    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e79149-3d3f-4037-acc4-047054a5f1eb" containerName="test-curl-7560-140479637038688"
Jul 24 16:28:02.534569 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.534555    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e79149-3d3f-4037-acc4-047054a5f1eb" containerName="test-curl-7560-140479637038688"
Jul 24 16:28:02.537312 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.537298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:02.547103 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.547078    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:02.596969 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.596936    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5wff\" (UniqueName: \"kubernetes.io/projected/35e79149-3d3f-4037-acc4-047054a5f1eb-kube-api-access-t5wff\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:02.697357 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.697319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwd9z\" (UniqueName: \"kubernetes.io/projected/3f2cc771-908c-49a4-a328-360d2e1fb1b1-kube-api-access-vwd9z\") pod \"test-curl-7560-140479637039696\" (UID: \"3f2cc771-908c-49a4-a328-360d2e1fb1b1\") " pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:02.798740 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.798642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd9z\" (UniqueName: \"kubernetes.io/projected/3f2cc771-908c-49a4-a328-360d2e1fb1b1-kube-api-access-vwd9z\") pod \"test-curl-7560-140479637039696\" (UID: \"3f2cc771-908c-49a4-a328-360d2e1fb1b1\") " pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:02.807592 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.807555    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwd9z\" (UniqueName: \"kubernetes.io/projected/3f2cc771-908c-49a4-a328-360d2e1fb1b1-kube-api-access-vwd9z\") pod \"test-curl-7560-140479637039696\" (UID: \"3f2cc771-908c-49a4-a328-360d2e1fb1b1\") " pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:02.847199 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.847167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:02.983481 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:02.983438    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:02.986659 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:28:02.986628    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f2cc771_908c_49a4_a328_360d2e1fb1b1.slice/crio-93e51866ffd91a78780fa17cc210e65c6137463f0c5b4872ebcda9e3c8b4f73d WatchSource:0}: Error finding container 93e51866ffd91a78780fa17cc210e65c6137463f0c5b4872ebcda9e3c8b4f73d: Status 404 returned error can't find the container with id 93e51866ffd91a78780fa17cc210e65c6137463f0c5b4872ebcda9e3c8b4f73d
Jul 24 16:28:03.005673 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.005648    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35e79149-3d3f-4037-acc4-047054a5f1eb" path="/var/lib/kubelet/pods/35e79149-3d3f-4037-acc4-047054a5f1eb/volumes"
Jul 24 16:28:03.108837 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.108747    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx" podUID="042fdd01-92eb-4004-8333-6baf81e1d2ea" containerName="main" containerID="cri-o://f66586ac9f808e07c390bb6b5a15eff9d5e07721bef013c22f47d3e25d694aeb" gracePeriod=45
Jul 24 16:28:03.251300 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.251258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479637039696" event={"ID":"3f2cc771-908c-49a4-a328-360d2e1fb1b1","Type":"ContainerStarted","Data":"93e51866ffd91a78780fa17cc210e65c6137463f0c5b4872ebcda9e3c8b4f73d"}
Jul 24 16:28:03.253054 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.253024    2576 generic.go:356] "Generic (PLEG): container finished" podID="042fdd01-92eb-4004-8333-6baf81e1d2ea" containerID="f66586ac9f808e07c390bb6b5a15eff9d5e07721bef013c22f47d3e25d694aeb" exitCode=0
Jul 24 16:28:03.253179 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.253090    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx" event={"ID":"042fdd01-92eb-4004-8333-6baf81e1d2ea","Type":"ContainerDied","Data":"f66586ac9f808e07c390bb6b5a15eff9d5e07721bef013c22f47d3e25d694aeb"}
Jul 24 16:28:03.254636 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.254618    2576 scope.go:117] "RemoveContainer" containerID="1d8034e39f6687e9d19496e69c9ac4e63b479f9af4ad8427a0d8f06cfd4f124b"
Jul 24 16:28:03.254636 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.254629    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637038688"
Jul 24 16:28:03.354740 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.354716    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:28:03.505937 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.505900    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-tmp-dir\") pod \"042fdd01-92eb-4004-8333-6baf81e1d2ea\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") "
Jul 24 16:28:03.505937 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.505938    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-model-cache\") pod \"042fdd01-92eb-4004-8333-6baf81e1d2ea\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") "
Jul 24 16:28:03.506507 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.505990    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-dshm\") pod \"042fdd01-92eb-4004-8333-6baf81e1d2ea\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") "
Jul 24 16:28:03.506507 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.506026    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65r9r\" (UniqueName: \"kubernetes.io/projected/042fdd01-92eb-4004-8333-6baf81e1d2ea-kube-api-access-65r9r\") pod \"042fdd01-92eb-4004-8333-6baf81e1d2ea\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") "
Jul 24 16:28:03.506507 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.506052    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042fdd01-92eb-4004-8333-6baf81e1d2ea-tls-certs\") pod \"042fdd01-92eb-4004-8333-6baf81e1d2ea\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") "
Jul 24 16:28:03.506507 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.506103    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-home\") pod \"042fdd01-92eb-4004-8333-6baf81e1d2ea\" (UID: \"042fdd01-92eb-4004-8333-6baf81e1d2ea\") "
Jul 24 16:28:03.506507 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.506135    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "042fdd01-92eb-4004-8333-6baf81e1d2ea" (UID: "042fdd01-92eb-4004-8333-6baf81e1d2ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:28:03.506764 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.506658    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-home" (OuterVolumeSpecName: "home") pod "042fdd01-92eb-4004-8333-6baf81e1d2ea" (UID: "042fdd01-92eb-4004-8333-6baf81e1d2ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:28:03.506764 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.506720    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "042fdd01-92eb-4004-8333-6baf81e1d2ea" (UID: "042fdd01-92eb-4004-8333-6baf81e1d2ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:28:03.506868 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.506786    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:03.506868 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.506805    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:03.509228 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.509196    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-dshm" (OuterVolumeSpecName: "dshm") pod "042fdd01-92eb-4004-8333-6baf81e1d2ea" (UID: "042fdd01-92eb-4004-8333-6baf81e1d2ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:28:03.509228 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.509215    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042fdd01-92eb-4004-8333-6baf81e1d2ea-kube-api-access-65r9r" (OuterVolumeSpecName: "kube-api-access-65r9r") pod "042fdd01-92eb-4004-8333-6baf81e1d2ea" (UID: "042fdd01-92eb-4004-8333-6baf81e1d2ea"). InnerVolumeSpecName "kube-api-access-65r9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:28:03.512425 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.512403    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/042fdd01-92eb-4004-8333-6baf81e1d2ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "042fdd01-92eb-4004-8333-6baf81e1d2ea" (UID: "042fdd01-92eb-4004-8333-6baf81e1d2ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:28:03.607346 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.607315    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:03.607511 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.607366    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65r9r\" (UniqueName: \"kubernetes.io/projected/042fdd01-92eb-4004-8333-6baf81e1d2ea-kube-api-access-65r9r\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:03.607511 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.607377    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042fdd01-92eb-4004-8333-6baf81e1d2ea-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:03.607511 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:03.607387    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042fdd01-92eb-4004-8333-6baf81e1d2ea-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:04.069337 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.069295    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq" podUID="d2b868d4-79c5-4e1f-b753-3c31ea907cb1" containerName="main" containerID="cri-o://013cda53a57900000ffb05482fee4ad5c2641da5cc82d2f4698611a7dd0bdc49" gracePeriod=45
Jul 24 16:28:04.262813 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.262785    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"
Jul 24 16:28:04.262973 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.262791    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx" event={"ID":"042fdd01-92eb-4004-8333-6baf81e1d2ea","Type":"ContainerDied","Data":"78c01177a308b3839e6d709cfe70a8d63a6030f15fcbdded102a8bacdb1d5de6"}
Jul 24 16:28:04.262973 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.262889    2576 scope.go:117] "RemoveContainer" containerID="f66586ac9f808e07c390bb6b5a15eff9d5e07721bef013c22f47d3e25d694aeb"
Jul 24 16:28:04.265763 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.265738    2576 generic.go:356] "Generic (PLEG): container finished" podID="d2b868d4-79c5-4e1f-b753-3c31ea907cb1" containerID="013cda53a57900000ffb05482fee4ad5c2641da5cc82d2f4698611a7dd0bdc49" exitCode=0
Jul 24 16:28:04.265905 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.265817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq" event={"ID":"d2b868d4-79c5-4e1f-b753-3c31ea907cb1","Type":"ContainerDied","Data":"013cda53a57900000ffb05482fee4ad5c2641da5cc82d2f4698611a7dd0bdc49"}
Jul 24 16:28:04.267697 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.267672    2576 generic.go:356] "Generic (PLEG): container finished" podID="3f2cc771-908c-49a4-a328-360d2e1fb1b1" containerID="c976e32b043672c24b89b4aa61b13e97e2f3e0c9d2a9d003369865e89b2201d6" exitCode=0
Jul 24 16:28:04.267808 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.267724    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479637039696" event={"ID":"3f2cc771-908c-49a4-a328-360d2e1fb1b1","Type":"ContainerDied","Data":"c976e32b043672c24b89b4aa61b13e97e2f3e0c9d2a9d003369865e89b2201d6"}
Jul 24 16:28:04.293429 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.293405    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"]
Jul 24 16:28:04.298971 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.298946    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ed0643/rate-test-model-ed0643-kserve-575d6c8b4c-d54lx"]
Jul 24 16:28:04.325115 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.325060    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:28:04.415098 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.415068    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-dshm\") pod \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") "
Jul 24 16:28:04.415283 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.415117    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-home\") pod \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") "
Jul 24 16:28:04.415283 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.415161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7tkm\" (UniqueName: \"kubernetes.io/projected/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-kube-api-access-k7tkm\") pod \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") "
Jul 24 16:28:04.415283 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.415185    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tls-certs\") pod \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") "
Jul 24 16:28:04.415283 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.415200    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tmp-dir\") pod \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") "
Jul 24 16:28:04.415283 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.415259    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-model-cache\") pod \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\" (UID: \"d2b868d4-79c5-4e1f-b753-3c31ea907cb1\") "
Jul 24 16:28:04.415572 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.415546    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2b868d4-79c5-4e1f-b753-3c31ea907cb1" (UID: "d2b868d4-79c5-4e1f-b753-3c31ea907cb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:28:04.415632 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.415583    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-home" (OuterVolumeSpecName: "home") pod "d2b868d4-79c5-4e1f-b753-3c31ea907cb1" (UID: "d2b868d4-79c5-4e1f-b753-3c31ea907cb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:28:04.415690 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.415653    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2b868d4-79c5-4e1f-b753-3c31ea907cb1" (UID: "d2b868d4-79c5-4e1f-b753-3c31ea907cb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:28:04.417514 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.417490    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-dshm" (OuterVolumeSpecName: "dshm") pod "d2b868d4-79c5-4e1f-b753-3c31ea907cb1" (UID: "d2b868d4-79c5-4e1f-b753-3c31ea907cb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:28:04.417629 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.417561    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2b868d4-79c5-4e1f-b753-3c31ea907cb1" (UID: "d2b868d4-79c5-4e1f-b753-3c31ea907cb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:28:04.417629 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.417585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-kube-api-access-k7tkm" (OuterVolumeSpecName: "kube-api-access-k7tkm") pod "d2b868d4-79c5-4e1f-b753-3c31ea907cb1" (UID: "d2b868d4-79c5-4e1f-b753-3c31ea907cb1"). InnerVolumeSpecName "kube-api-access-k7tkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:28:04.516322 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.516291    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:04.516322 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.516320    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7tkm\" (UniqueName: \"kubernetes.io/projected/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-kube-api-access-k7tkm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:04.516322 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.516332    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:04.516761 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.516343    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:04.516761 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.516351    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:04.516761 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:04.516360    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b868d4-79c5-4e1f-b753-3c31ea907cb1-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:05.005385 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.005348    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="042fdd01-92eb-4004-8333-6baf81e1d2ea" path="/var/lib/kubelet/pods/042fdd01-92eb-4004-8333-6baf81e1d2ea/volumes"
Jul 24 16:28:05.273895 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.273809    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"
Jul 24 16:28:05.273895 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.273854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq" event={"ID":"d2b868d4-79c5-4e1f-b753-3c31ea907cb1","Type":"ContainerDied","Data":"77251ecfc8f5f723fc3262e7dc7963bce055201fe446bb3656b1ae745df5e97f"}
Jul 24 16:28:05.274166 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.273907    2576 scope.go:117] "RemoveContainer" containerID="013cda53a57900000ffb05482fee4ad5c2641da5cc82d2f4698611a7dd0bdc49"
Jul 24 16:28:05.289803 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.289764    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"]
Jul 24 16:28:05.294044 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.294015    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5aebfb/rate-test-model-5aebfb-kserve-7d7ffb68ff-6p2cq"]
Jul 24 16:28:05.417428 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.417397    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:05.437455 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.437435    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7560-140479637039696_3f2cc771-908c-49a4-a328-360d2e1fb1b1/test-curl-7560-140479637039696/0.log"
Jul 24 16:28:05.461261 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.461215    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:05.463076 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.463058    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:05.524460 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.524378    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwd9z\" (UniqueName: \"kubernetes.io/projected/3f2cc771-908c-49a4-a328-360d2e1fb1b1-kube-api-access-vwd9z\") pod \"3f2cc771-908c-49a4-a328-360d2e1fb1b1\" (UID: \"3f2cc771-908c-49a4-a328-360d2e1fb1b1\") "
Jul 24 16:28:05.526628 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.526587    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2cc771-908c-49a4-a328-360d2e1fb1b1-kube-api-access-vwd9z" (OuterVolumeSpecName: "kube-api-access-vwd9z") pod "3f2cc771-908c-49a4-a328-360d2e1fb1b1" (UID: "3f2cc771-908c-49a4-a328-360d2e1fb1b1"). InnerVolumeSpecName "kube-api-access-vwd9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:28:05.584198 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584164    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:05.584506 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584494    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2cc771-908c-49a4-a328-360d2e1fb1b1" containerName="test-curl-7560-140479637039696"
Jul 24 16:28:05.584563 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584509    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2cc771-908c-49a4-a328-360d2e1fb1b1" containerName="test-curl-7560-140479637039696"
Jul 24 16:28:05.584563 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584528    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="042fdd01-92eb-4004-8333-6baf81e1d2ea" containerName="main"
Jul 24 16:28:05.584563 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584534    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="042fdd01-92eb-4004-8333-6baf81e1d2ea" containerName="main"
Jul 24 16:28:05.584563 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584542    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b868d4-79c5-4e1f-b753-3c31ea907cb1" containerName="main"
Jul 24 16:28:05.584563 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584547    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b868d4-79c5-4e1f-b753-3c31ea907cb1" containerName="main"
Jul 24 16:28:05.584707 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584603    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b868d4-79c5-4e1f-b753-3c31ea907cb1" containerName="main"
Jul 24 16:28:05.584707 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584612    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="042fdd01-92eb-4004-8333-6baf81e1d2ea" containerName="main"
Jul 24 16:28:05.584707 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.584619    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2cc771-908c-49a4-a328-360d2e1fb1b1" containerName="test-curl-7560-140479637039696"
Jul 24 16:28:05.587465 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.587450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:05.593415 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.593388    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:05.625222 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.625192    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwd9z\" (UniqueName: \"kubernetes.io/projected/3f2cc771-908c-49a4-a328-360d2e1fb1b1-kube-api-access-vwd9z\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:05.726211 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.726178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln85t\" (UniqueName: \"kubernetes.io/projected/5b1f9a10-c343-4b31-910c-00b747468c49-kube-api-access-ln85t\") pod \"test-curl-7560-140479637039696\" (UID: \"5b1f9a10-c343-4b31-910c-00b747468c49\") " pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:05.827032 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.826937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln85t\" (UniqueName: \"kubernetes.io/projected/5b1f9a10-c343-4b31-910c-00b747468c49-kube-api-access-ln85t\") pod \"test-curl-7560-140479637039696\" (UID: \"5b1f9a10-c343-4b31-910c-00b747468c49\") " pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:05.835627 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.835595    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln85t\" (UniqueName: \"kubernetes.io/projected/5b1f9a10-c343-4b31-910c-00b747468c49-kube-api-access-ln85t\") pod \"test-curl-7560-140479637039696\" (UID: \"5b1f9a10-c343-4b31-910c-00b747468c49\") " pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:05.897404 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:05.897367    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:06.058991 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:06.058966    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:06.060528 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:28:06.060500    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b1f9a10_c343_4b31_910c_00b747468c49.slice/crio-5a51b1f54bb2b6b66cf1c61225526ab1d6d9a50b7ff3549f0b30793992d97d37 WatchSource:0}: Error finding container 5a51b1f54bb2b6b66cf1c61225526ab1d6d9a50b7ff3549f0b30793992d97d37: Status 404 returned error can't find the container with id 5a51b1f54bb2b6b66cf1c61225526ab1d6d9a50b7ff3549f0b30793992d97d37
Jul 24 16:28:06.281561 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:06.281521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479637039696" event={"ID":"5b1f9a10-c343-4b31-910c-00b747468c49","Type":"ContainerStarted","Data":"5a51b1f54bb2b6b66cf1c61225526ab1d6d9a50b7ff3549f0b30793992d97d37"}
Jul 24 16:28:06.282705 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:06.282682    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:06.282842 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:06.282724    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93e51866ffd91a78780fa17cc210e65c6137463f0c5b4872ebcda9e3c8b4f73d"
Jul 24 16:28:06.294549 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:06.294526    2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7560-140479637039696" oldPodUID="3f2cc771-908c-49a4-a328-360d2e1fb1b1" podUID="5b1f9a10-c343-4b31-910c-00b747468c49"
Jul 24 16:28:07.005862 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:07.005833    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f2cc771-908c-49a4-a328-360d2e1fb1b1" path="/var/lib/kubelet/pods/3f2cc771-908c-49a4-a328-360d2e1fb1b1/volumes"
Jul 24 16:28:07.006260 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:07.006206    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b868d4-79c5-4e1f-b753-3c31ea907cb1" path="/var/lib/kubelet/pods/d2b868d4-79c5-4e1f-b753-3c31ea907cb1/volumes"
Jul 24 16:28:07.289965 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:07.289878    2576 generic.go:356] "Generic (PLEG): container finished" podID="5b1f9a10-c343-4b31-910c-00b747468c49" containerID="f72b95ab53564949242d793539ffb45b7e1b6f1e977a461888e967c389a82fa8" exitCode=0
Jul 24 16:28:07.290113 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:07.289963    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-140479637039696" event={"ID":"5b1f9a10-c343-4b31-910c-00b747468c49","Type":"ContainerDied","Data":"f72b95ab53564949242d793539ffb45b7e1b6f1e977a461888e967c389a82fa8"}
Jul 24 16:28:08.436671 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:08.436646    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:08.457260 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:08.457217    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7560-140479637039696_5b1f9a10-c343-4b31-910c-00b747468c49/test-curl-7560-140479637039696/0.log"
Jul 24 16:28:08.482131 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:08.482100    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:08.485621 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:08.485589    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7560-140479637039696"]
Jul 24 16:28:08.550016 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:08.549982    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln85t\" (UniqueName: \"kubernetes.io/projected/5b1f9a10-c343-4b31-910c-00b747468c49-kube-api-access-ln85t\") pod \"5b1f9a10-c343-4b31-910c-00b747468c49\" (UID: \"5b1f9a10-c343-4b31-910c-00b747468c49\") "
Jul 24 16:28:08.552206 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:08.552146    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1f9a10-c343-4b31-910c-00b747468c49-kube-api-access-ln85t" (OuterVolumeSpecName: "kube-api-access-ln85t") pod "5b1f9a10-c343-4b31-910c-00b747468c49" (UID: "5b1f9a10-c343-4b31-910c-00b747468c49"). InnerVolumeSpecName "kube-api-access-ln85t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:28:08.651470 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:08.651420    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln85t\" (UniqueName: \"kubernetes.io/projected/5b1f9a10-c343-4b31-910c-00b747468c49-kube-api-access-ln85t\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:28:09.005334 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:09.005304    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1f9a10-c343-4b31-910c-00b747468c49" path="/var/lib/kubelet/pods/5b1f9a10-c343-4b31-910c-00b747468c49/volumes"
Jul 24 16:28:09.306896 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:09.306818    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-140479637039696"
Jul 24 16:28:09.306896 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:09.306851    2576 scope.go:117] "RemoveContainer" containerID="f72b95ab53564949242d793539ffb45b7e1b6f1e977a461888e967c389a82fa8"
Jul 24 16:28:20.560222 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:20.560186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:28:27.653776 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.653735    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"]
Jul 24 16:28:27.654173 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.654063    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1f9a10-c343-4b31-910c-00b747468c49" containerName="test-curl-7560-140479637039696"
Jul 24 16:28:27.654173 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.654075    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1f9a10-c343-4b31-910c-00b747468c49" containerName="test-curl-7560-140479637039696"
Jul 24 16:28:27.654173 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.654137    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1f9a10-c343-4b31-910c-00b747468c49" containerName="test-curl-7560-140479637039696"
Jul 24 16:28:27.659192 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.659170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.661611 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.661589    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2c8631-openshift-default-dockercfg-hqx98\""
Jul 24 16:28:27.661611 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.661603    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 16:28:27.669946 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.669923    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"]
Jul 24 16:28:27.714051 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.714011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-envoy\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.714051 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.714053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-token\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.714299 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.714133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-socket\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.714299 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.714171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-certs\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.714299 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.714192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9cpr\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-kube-api-access-b9cpr\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.714299 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.714218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-data\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.714443 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.714316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-podinfo\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.714443 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.714347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-credential-socket\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.714443 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.714408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istiod-ca-cert\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815194 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istiod-ca-cert\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815194 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-envoy\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815473 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-token\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815473 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-socket\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815473 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-certs\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815473 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815360    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9cpr\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-kube-api-access-b9cpr\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815473 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-data\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815747 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-podinfo\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815747 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-credential-socket\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815863 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-socket\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.815956 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815934    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-data\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.816078 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-certs\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.816078 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.815965    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istiod-ca-cert\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.816078 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.816025    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-credential-socket\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.817887 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.817865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-envoy\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.818115 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.818096    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-podinfo\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.823360 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.823335    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-token\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.823538 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.823521    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9cpr\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-kube-api-access-b9cpr\") pod \"e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") " pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:27.970966 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:27.970861    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:28.111781 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:28.111628    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"]
Jul 24 16:28:28.113963 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:28:28.113936    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8fb632_f838_43d2_9b0f_85c4d07acc77.slice/crio-786dd5a4f2b51ccdcddb4c94c75e2e5cc81f8219fdc901b25742fc7747739159 WatchSource:0}: Error finding container 786dd5a4f2b51ccdcddb4c94c75e2e5cc81f8219fdc901b25742fc7747739159: Status 404 returned error can't find the container with id 786dd5a4f2b51ccdcddb4c94c75e2e5cc81f8219fdc901b25742fc7747739159
Jul 24 16:28:28.409304 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:28.409268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" event={"ID":"5b8fb632-f838-43d2-9b0f-85c4d07acc77","Type":"ContainerStarted","Data":"786dd5a4f2b51ccdcddb4c94c75e2e5cc81f8219fdc901b25742fc7747739159"}
Jul 24 16:28:30.706450 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:30.706411    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:28:30.706835 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:30.706500    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:28:30.706835 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:30.706547    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:28:31.435984 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:31.435946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" event={"ID":"5b8fb632-f838-43d2-9b0f-85c4d07acc77","Type":"ContainerStarted","Data":"887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0"}
Jul 24 16:28:31.460960 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:31.460911    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" podStartSLOduration=1.8707065859999998 podStartE2EDuration="4.460896724s" totalImagesPullingTime="2.590190138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:28:31.459481864 +0000 UTC m=+2870.944042464" watchObservedRunningTime="2026-07-24 16:28:31.460896724 +0000 UTC m=+2870.945457322"
Jul 24 16:28:31.971668 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:31.971629    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:31.976781 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:31.976756    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:32.441064 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:32.441025    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:32.441915 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:32.441895    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:38.566784 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:38.566748    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:28:46.519406 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:46.519371    2576 generic.go:356] "Generic (PLEG): container finished" podID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerID="887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0" exitCode=0
Jul 24 16:28:46.519944 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:46.519448    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" event={"ID":"5b8fb632-f838-43d2-9b0f-85c4d07acc77","Type":"ContainerDied","Data":"887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0"}
Jul 24 16:28:46.520023 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:46.519946    2576 scope.go:117] "RemoveContainer" containerID="887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0"
Jul 24 16:28:46.521070 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:46.521040    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:28:46.521148 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:46.521110    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:28:46.521148 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:46.521142    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:28:47.441434 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:47.441388    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:47.526474 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:47.526439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" event={"ID":"5b8fb632-f838-43d2-9b0f-85c4d07acc77","Type":"ContainerStarted","Data":"55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557"}
Jul 24 16:28:47.971556 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:47.971525    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:47.972774 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:47.972742    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 24 16:28:47.972926 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:47.972798    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 24 16:28:48.971921 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:48.971880    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 24 16:28:48.972356 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:48.971943    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 24 16:28:49.971529 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:49.971482    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 24 16:28:49.971742 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:49.971556    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 24 16:28:50.971976 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:50.971925    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 24 16:28:50.972461 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:50.971995    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 24 16:28:51.975400 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:51.975369    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:51.975777 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:51.975635    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:28:51.976379 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:28:51.976365    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:29:03.375643 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.375606    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"]
Jul 24 16:29:03.380104 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.380081    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.383545 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.383522    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2c5dfe\"/\"kube-root-ca.crt\""
Jul 24 16:29:03.384034 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.384017    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2c5dfe\"/\"ipp-route-2c5dfe-kserve-self-signed-certs\""
Jul 24 16:29:03.384733 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.384713    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2c5dfe\"/\"default-dockercfg-dddmp\""
Jul 24 16:29:03.384867 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.384783    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2c5dfe\"/\"openshift-service-ca.crt\""
Jul 24 16:29:03.393450 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.393427    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"]
Jul 24 16:29:03.507759 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.507723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zg8\" (UniqueName: \"kubernetes.io/projected/386387b5-414d-4049-959d-0526224025d6-kube-api-access-x5zg8\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.507759 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.507762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-home\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.508033 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.507860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-tmp-dir\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.508033 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.507892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386387b5-414d-4049-959d-0526224025d6-tls-certs\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.508033 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.507926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-model-cache\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.508033 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.508004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-dshm\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.609328 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.609292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-dshm\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.609328 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.609331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zg8\" (UniqueName: \"kubernetes.io/projected/386387b5-414d-4049-959d-0526224025d6-kube-api-access-x5zg8\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.609574 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.609353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-home\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.609574 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.609397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-tmp-dir\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.609574 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.609415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386387b5-414d-4049-959d-0526224025d6-tls-certs\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.609574 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.609439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-model-cache\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.609821 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.609801    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-tmp-dir\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.609877 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.609857    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-home\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.609914 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.609882    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-model-cache\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.611903 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.611880    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-dshm\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.612047 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.612029    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386387b5-414d-4049-959d-0526224025d6-tls-certs\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.617872 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.617847    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zg8\" (UniqueName: \"kubernetes.io/projected/386387b5-414d-4049-959d-0526224025d6-kube-api-access-x5zg8\") pod \"ipp-route-2c5dfe-kserve-56b848c9-wj8bw\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") " pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.691160 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.691122    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:03.827576 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:03.827550    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"]
Jul 24 16:29:03.829352 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:29:03.829313    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod386387b5_414d_4049_959d_0526224025d6.slice/crio-d17f4d6dd6e77b800052a82b344558e347f8d7a80bac82466a5c30e10509fcfe WatchSource:0}: Error finding container d17f4d6dd6e77b800052a82b344558e347f8d7a80bac82466a5c30e10509fcfe: Status 404 returned error can't find the container with id d17f4d6dd6e77b800052a82b344558e347f8d7a80bac82466a5c30e10509fcfe
Jul 24 16:29:04.623632 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:04.623598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw" event={"ID":"386387b5-414d-4049-959d-0526224025d6","Type":"ContainerStarted","Data":"0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6"}
Jul 24 16:29:04.623632 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:04.623644    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw" event={"ID":"386387b5-414d-4049-959d-0526224025d6","Type":"ContainerStarted","Data":"d17f4d6dd6e77b800052a82b344558e347f8d7a80bac82466a5c30e10509fcfe"}
Jul 24 16:29:04.640988 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:04.640943    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw" podStartSLOduration=1.640928519 podStartE2EDuration="1.640928519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:29:04.639574474 +0000 UTC m=+2904.124135085" watchObservedRunningTime="2026-07-24 16:29:04.640928519 +0000 UTC m=+2904.125489127"
Jul 24 16:29:07.665374 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:07.660476    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:29:13.691783 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:13.691746    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:13.692314 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:13.691801    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:13.704391 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:13.704366    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:14.690514 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:14.690478    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:29:43.457141 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:43.457104    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:29:55.227787 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.227746    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"]
Jul 24 16:29:55.231217 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.231193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.233866 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.233847    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-4c4c5c-openshift-default-dockercfg-9mp8s\""
Jul 24 16:29:55.248064 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.248041    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"]
Jul 24 16:29:55.364799 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.364744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-socket\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.364799 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.364797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-data\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.364799 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.364815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-istio-token\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.365068 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.364896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506dbf6e-1678-4e42-b688-b859450a9b47-istio-podinfo\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.365068 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.364941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-certs\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.365068 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.365003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-credential-socket\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.365068 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.365051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-envoy\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.365198 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.365070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmsx\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-kube-api-access-sdmsx\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.365198 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.365089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506dbf6e-1678-4e42-b688-b859450a9b47-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.465948 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.465893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-credential-socket\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.465948 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.465950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-envoy\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466177 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmsx\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-kube-api-access-sdmsx\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466177 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506dbf6e-1678-4e42-b688-b859450a9b47-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466177 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466144    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-socket\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466366 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-data\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466366 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-istio-token\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466471 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466389    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-credential-socket\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466471 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506dbf6e-1678-4e42-b688-b859450a9b47-istio-podinfo\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466471 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-socket\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466471 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-certs\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466673 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466631    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-data\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466825 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506dbf6e-1678-4e42-b688-b859450a9b47-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.466887 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.466811    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-certs\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.468454 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.468433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-envoy\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.468829 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.468812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506dbf6e-1678-4e42-b688-b859450a9b47-istio-podinfo\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.474934 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.474906    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-istio-token\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.475058 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.474980    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmsx\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-kube-api-access-sdmsx\") pod \"e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") " pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.541289 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.541176    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:55.683076 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.683053    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"]
Jul 24 16:29:55.685099 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:29:55.685071    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506dbf6e_1678_4e42_b688_b859450a9b47.slice/crio-c08f6d90a7cece93e7f43a48e4831117d49cefd19569eca8719f0912a1bd934f WatchSource:0}: Error finding container c08f6d90a7cece93e7f43a48e4831117d49cefd19569eca8719f0912a1bd934f: Status 404 returned error can't find the container with id c08f6d90a7cece93e7f43a48e4831117d49cefd19569eca8719f0912a1bd934f
Jul 24 16:29:55.687579 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.687546    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:29:55.687683 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.687631    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:29:55.687746 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.687678    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:29:55.897215 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.897183    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" event={"ID":"506dbf6e-1678-4e42-b688-b859450a9b47","Type":"ContainerStarted","Data":"a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97"}
Jul 24 16:29:55.897407 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.897235    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" event={"ID":"506dbf6e-1678-4e42-b688-b859450a9b47","Type":"ContainerStarted","Data":"c08f6d90a7cece93e7f43a48e4831117d49cefd19569eca8719f0912a1bd934f"}
Jul 24 16:29:55.918103 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:55.918056    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" podStartSLOduration=0.918039256 podStartE2EDuration="918.039256ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:29:55.91567809 +0000 UTC m=+2955.400238711" watchObservedRunningTime="2026-07-24 16:29:55.918039256 +0000 UTC m=+2955.402599854"
Jul 24 16:29:56.542320 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:56.542285    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:56.547157 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:56.547134    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:56.902776 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:56.902743    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:29:56.903676 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:29:56.903659    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:30:00.142725 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.142689    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz"]
Jul 24 16:30:00.146335 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.146312    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb"]
Jul 24 16:30:00.146492 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.146430    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz"
Jul 24 16:30:00.149267 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.149226    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\""
Jul 24 16:30:00.149549 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.149535    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5"]
Jul 24 16:30:00.149652 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.149639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb"
Jul 24 16:30:00.152532 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.152513    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk"]
Jul 24 16:30:00.152638 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.152625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5"
Jul 24 16:30:00.156121 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.156102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk"
Jul 24 16:30:00.160527 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.160509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz"
Jul 24 16:30:00.161286 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.161269    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz"]
Jul 24 16:30:00.164497 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.164479    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb"]
Jul 24 16:30:00.167668 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.167647    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5"]
Jul 24 16:30:00.170704 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.170674    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk"]
Jul 24 16:30:00.173163 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.173146    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb"
Jul 24 16:30:00.173831 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.173812    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"]
Jul 24 16:30:00.180985 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.180963    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"]
Jul 24 16:30:00.181227 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.181179    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"
Jul 24 16:30:00.187633 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.187612    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5"
Jul 24 16:30:00.236028 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.235997    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk"
Jul 24 16:30:00.247228 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.247196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"
Jul 24 16:30:00.486467 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.486446    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"]
Jul 24 16:30:00.488699 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:30:00.488670    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4671a1f7_a63a_4cb8_95b6_d6a5e9911efa.slice/crio-dd5537647006c7327d4954b4764dfcd35edd8581edb8b759293a3b842e9d800f WatchSource:0}: Error finding container dd5537647006c7327d4954b4764dfcd35edd8581edb8b759293a3b842e9d800f: Status 404 returned error can't find the container with id dd5537647006c7327d4954b4764dfcd35edd8581edb8b759293a3b842e9d800f
Jul 24 16:30:00.570707 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.570673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz"]
Jul 24 16:30:00.572186 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:30:00.572163    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda677a159_699f_4f73_ae05_3738e47ea5a7.slice/crio-1c2c83d22dd7b76fccbcc174d74f50e260752e5f861189dd44ff6ecd178b13b1 WatchSource:0}: Error finding container 1c2c83d22dd7b76fccbcc174d74f50e260752e5f861189dd44ff6ecd178b13b1: Status 404 returned error can't find the container with id 1c2c83d22dd7b76fccbcc174d74f50e260752e5f861189dd44ff6ecd178b13b1
Jul 24 16:30:00.629717 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.629692    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5"]
Jul 24 16:30:00.631433 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:30:00.631406    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec79287b_3d47_4fc9_a71d_b7d988d91b2b.slice/crio-5f802f4cd2b7d09e275163898cb000b4b4193a7dfcf1d60f779faec0c02a51a3 WatchSource:0}: Error finding container 5f802f4cd2b7d09e275163898cb000b4b4193a7dfcf1d60f779faec0c02a51a3: Status 404 returned error can't find the container with id 5f802f4cd2b7d09e275163898cb000b4b4193a7dfcf1d60f779faec0c02a51a3
Jul 24 16:30:00.636768 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.636747    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb"]
Jul 24 16:30:00.638037 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:30:00.638008    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae09dfd_795f_4ea5_a50d_f8d6087d798e.slice/crio-2be53f1714e77005f67c4676dde72cf1a4c2bd7d1e70d9c167e316f67c9e8ed1 WatchSource:0}: Error finding container 2be53f1714e77005f67c4676dde72cf1a4c2bd7d1e70d9c167e316f67c9e8ed1: Status 404 returned error can't find the container with id 2be53f1714e77005f67c4676dde72cf1a4c2bd7d1e70d9c167e316f67c9e8ed1
Jul 24 16:30:00.670722 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:30:00.670692    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16571df9_8318_4691_a55f_089e5eaf786a.slice/crio-eba60560d3bb6440ca6f7ba097d5b3d1a47905adee5836206ca6b2919537db0d WatchSource:0}: Error finding container eba60560d3bb6440ca6f7ba097d5b3d1a47905adee5836206ca6b2919537db0d: Status 404 returned error can't find the container with id eba60560d3bb6440ca6f7ba097d5b3d1a47905adee5836206ca6b2919537db0d
Jul 24 16:30:00.671290 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.671227    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk"]
Jul 24 16:30:00.925094 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.925004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb" event={"ID":"7ae09dfd-795f-4ea5-a50d-f8d6087d798e","Type":"ContainerStarted","Data":"2be53f1714e77005f67c4676dde72cf1a4c2bd7d1e70d9c167e316f67c9e8ed1"}
Jul 24 16:30:00.926303 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.926276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z" event={"ID":"4671a1f7-a63a-4cb8-95b6-d6a5e9911efa","Type":"ContainerStarted","Data":"dd5537647006c7327d4954b4764dfcd35edd8581edb8b759293a3b842e9d800f"}
Jul 24 16:30:00.927464 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.927436    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5" event={"ID":"ec79287b-3d47-4fc9-a71d-b7d988d91b2b","Type":"ContainerStarted","Data":"5f802f4cd2b7d09e275163898cb000b4b4193a7dfcf1d60f779faec0c02a51a3"}
Jul 24 16:30:00.928525 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.928501    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz" event={"ID":"a677a159-699f-4f73-ae05-3738e47ea5a7","Type":"ContainerStarted","Data":"1c2c83d22dd7b76fccbcc174d74f50e260752e5f861189dd44ff6ecd178b13b1"}
Jul 24 16:30:00.929710 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:00.929691    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk" event={"ID":"16571df9-8318-4691-a55f-089e5eaf786a","Type":"ContainerStarted","Data":"eba60560d3bb6440ca6f7ba097d5b3d1a47905adee5836206ca6b2919537db0d"}
Jul 24 16:30:03.470878 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:03.470847    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:30:03.955036 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:03.954994    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb" event={"ID":"7ae09dfd-795f-4ea5-a50d-f8d6087d798e","Type":"ContainerStarted","Data":"535e2bca02d5305964ac038d7641c7496b28cc972b789eb7c877bb18598728ae"}
Jul 24 16:30:03.956481 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:03.956454    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z" event={"ID":"4671a1f7-a63a-4cb8-95b6-d6a5e9911efa","Type":"ContainerStarted","Data":"f84072f18b4328a6c3f54f13c28ba07221ae0617da780cc6a2b7d5249b35ed87"}
Jul 24 16:30:03.957795 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:03.957771    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5" event={"ID":"ec79287b-3d47-4fc9-a71d-b7d988d91b2b","Type":"ContainerStarted","Data":"98ce0516de7f23b7716c186b27e6b9cce490774c028b879e00be21306b834a59"}
Jul 24 16:30:03.959226 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:03.959207    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz" event={"ID":"a677a159-699f-4f73-ae05-3738e47ea5a7","Type":"ContainerStarted","Data":"9cb9fb46dbcedb49372b8a9e958899e1856b472d39b8fe18a5c183a4d36837d6"}
Jul 24 16:30:03.960696 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:03.960678    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk" event={"ID":"16571df9-8318-4691-a55f-089e5eaf786a","Type":"ContainerStarted","Data":"fa7fa59f1099b900a020617bee123ea09e52dd29c956175421a502a33db317a8"}
Jul 24 16:30:03.970554 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:03.970512    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb" podStartSLOduration=1.299924745 podStartE2EDuration="3.970501041s" totalImagesPullingTime="2.670576296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:30:03.968759209 +0000 UTC m=+2963.453319808" watchObservedRunningTime="2026-07-24 16:30:03.970501041 +0000 UTC m=+2963.455061639"
Jul 24 16:30:03.983309 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:03.983270    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz" podStartSLOduration=1.1570458000000001 podStartE2EDuration="3.983257122s" totalImagesPullingTime="2.826211322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:30:03.981524757 +0000 UTC m=+2963.466085354" watchObservedRunningTime="2026-07-24 16:30:03.983257122 +0000 UTC m=+2963.467817714"
Jul 24 16:30:03.996687 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:03.996639    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5" podStartSLOduration=1.323593161 podStartE2EDuration="3.996627261s" totalImagesPullingTime="2.6730341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:30:03.994653613 +0000 UTC m=+2963.479214214" watchObservedRunningTime="2026-07-24 16:30:03.996627261 +0000 UTC m=+2963.481187877"
Jul 24 16:30:04.009881 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.009836    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk" podStartSLOduration=1.310782446 podStartE2EDuration="4.009825409s" totalImagesPullingTime="2.699042963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:30:04.008175243 +0000 UTC m=+2963.492735852" watchObservedRunningTime="2026-07-24 16:30:04.009825409 +0000 UTC m=+2963.494386007"
Jul 24 16:30:04.022804 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.022757    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z" podStartSLOduration=1.118834903 podStartE2EDuration="4.022744175s" totalImagesPullingTime="2.903909272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:30:04.021747563 +0000 UTC m=+2963.506308162" watchObservedRunningTime="2026-07-24 16:30:04.022744175 +0000 UTC m=+2963.507304772"
Jul 24 16:30:04.968033 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.967996    2576 generic.go:356] "Generic (PLEG): container finished" podID="16571df9-8318-4691-a55f-089e5eaf786a" containerID="fa7fa59f1099b900a020617bee123ea09e52dd29c956175421a502a33db317a8" exitCode=0
Jul 24 16:30:04.968534 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.968081    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk" event={"ID":"16571df9-8318-4691-a55f-089e5eaf786a","Type":"ContainerDied","Data":"fa7fa59f1099b900a020617bee123ea09e52dd29c956175421a502a33db317a8"}
Jul 24 16:30:04.969625 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.969597    2576 generic.go:356] "Generic (PLEG): container finished" podID="7ae09dfd-795f-4ea5-a50d-f8d6087d798e" containerID="535e2bca02d5305964ac038d7641c7496b28cc972b789eb7c877bb18598728ae" exitCode=0
Jul 24 16:30:04.969769 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.969669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb" event={"ID":"7ae09dfd-795f-4ea5-a50d-f8d6087d798e","Type":"ContainerDied","Data":"535e2bca02d5305964ac038d7641c7496b28cc972b789eb7c877bb18598728ae"}
Jul 24 16:30:04.971033 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.971005    2576 generic.go:356] "Generic (PLEG): container finished" podID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerID="f84072f18b4328a6c3f54f13c28ba07221ae0617da780cc6a2b7d5249b35ed87" exitCode=7
Jul 24 16:30:04.971141 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.971074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z" event={"ID":"4671a1f7-a63a-4cb8-95b6-d6a5e9911efa","Type":"ContainerDied","Data":"f84072f18b4328a6c3f54f13c28ba07221ae0617da780cc6a2b7d5249b35ed87"}
Jul 24 16:30:04.971268 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.971231    2576 scope.go:117] "RemoveContainer" containerID="f84072f18b4328a6c3f54f13c28ba07221ae0617da780cc6a2b7d5249b35ed87"
Jul 24 16:30:04.972735 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.972711    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec79287b-3d47-4fc9-a71d-b7d988d91b2b" containerID="98ce0516de7f23b7716c186b27e6b9cce490774c028b879e00be21306b834a59" exitCode=0
Jul 24 16:30:04.972854 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.972802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5" event={"ID":"ec79287b-3d47-4fc9-a71d-b7d988d91b2b","Type":"ContainerDied","Data":"98ce0516de7f23b7716c186b27e6b9cce490774c028b879e00be21306b834a59"}
Jul 24 16:30:04.974260 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.974155    2576 generic.go:356] "Generic (PLEG): container finished" podID="a677a159-699f-4f73-ae05-3738e47ea5a7" containerID="9cb9fb46dbcedb49372b8a9e958899e1856b472d39b8fe18a5c183a4d36837d6" exitCode=0
Jul 24 16:30:04.974260 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:04.974182    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz" event={"ID":"a677a159-699f-4f73-ae05-3738e47ea5a7","Type":"ContainerDied","Data":"9cb9fb46dbcedb49372b8a9e958899e1856b472d39b8fe18a5c183a4d36837d6"}
Jul 24 16:30:05.981292 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:05.981257    2576 generic.go:356] "Generic (PLEG): container finished" podID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerID="f8b82adbf619ef72010339f85a283ef9b18397865f0977f79ded3cfa628daf28" exitCode=7
Jul 24 16:30:05.981766 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:05.981348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z" event={"ID":"4671a1f7-a63a-4cb8-95b6-d6a5e9911efa","Type":"ContainerDied","Data":"f8b82adbf619ef72010339f85a283ef9b18397865f0977f79ded3cfa628daf28"}
Jul 24 16:30:05.981766 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:05.981424    2576 scope.go:117] "RemoveContainer" containerID="f84072f18b4328a6c3f54f13c28ba07221ae0617da780cc6a2b7d5249b35ed87"
Jul 24 16:30:05.981766 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:05.981488    2576 scope.go:117] "RemoveContainer" containerID="f8b82adbf619ef72010339f85a283ef9b18397865f0977f79ded3cfa628daf28"
Jul 24 16:30:05.981766 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:30:05.981744    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z_odh-ai-gateway-infra(4671a1f7-a63a-4cb8-95b6-d6a5e9911efa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa"
Jul 24 16:30:06.208904 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.208881    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk"
Jul 24 16:30:06.211204 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.211183    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb"
Jul 24 16:30:06.226934 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.226915    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5"
Jul 24 16:30:06.234070 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.234046    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz"
Jul 24 16:30:06.988599 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.988568    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb"
Jul 24 16:30:06.988599 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.988576    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb" event={"ID":"7ae09dfd-795f-4ea5-a50d-f8d6087d798e","Type":"ContainerDied","Data":"2be53f1714e77005f67c4676dde72cf1a4c2bd7d1e70d9c167e316f67c9e8ed1"}
Jul 24 16:30:06.989091 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.988653    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be53f1714e77005f67c4676dde72cf1a4c2bd7d1e70d9c167e316f67c9e8ed1"
Jul 24 16:30:06.990498 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.990477    2576 scope.go:117] "RemoveContainer" containerID="f8b82adbf619ef72010339f85a283ef9b18397865f0977f79ded3cfa628daf28"
Jul 24 16:30:06.990718 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:30:06.990693    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z_odh-ai-gateway-infra(4671a1f7-a63a-4cb8-95b6-d6a5e9911efa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa"
Jul 24 16:30:06.991653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.991635    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5"
Jul 24 16:30:06.991769 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.991661    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5" event={"ID":"ec79287b-3d47-4fc9-a71d-b7d988d91b2b","Type":"ContainerDied","Data":"5f802f4cd2b7d09e275163898cb000b4b4193a7dfcf1d60f779faec0c02a51a3"}
Jul 24 16:30:06.991769 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.991696    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f802f4cd2b7d09e275163898cb000b4b4193a7dfcf1d60f779faec0c02a51a3"
Jul 24 16:30:06.993060 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.993039    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz"
Jul 24 16:30:06.993176 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.993067    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748510-9whcz" event={"ID":"a677a159-699f-4f73-ae05-3738e47ea5a7","Type":"ContainerDied","Data":"1c2c83d22dd7b76fccbcc174d74f50e260752e5f861189dd44ff6ecd178b13b1"}
Jul 24 16:30:06.993176 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.993110    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c2c83d22dd7b76fccbcc174d74f50e260752e5f861189dd44ff6ecd178b13b1"
Jul 24 16:30:06.994427 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.994410    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk"
Jul 24 16:30:06.994526 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.994404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk" event={"ID":"16571df9-8318-4691-a55f-089e5eaf786a","Type":"ContainerDied","Data":"eba60560d3bb6440ca6f7ba097d5b3d1a47905adee5836206ca6b2919537db0d"}
Jul 24 16:30:06.994598 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:06.994535    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eba60560d3bb6440ca6f7ba097d5b3d1a47905adee5836206ca6b2919537db0d"
Jul 24 16:30:11.904891 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:11.904861    2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 24 16:30:11.905326 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:11.904941    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 24 16:30:12.023005 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:12.022937    2576 generic.go:356] "Generic (PLEG): container finished" podID="506dbf6e-1678-4e42-b688-b859450a9b47" containerID="a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97" exitCode=0
Jul 24 16:30:12.023140 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:12.023015    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" event={"ID":"506dbf6e-1678-4e42-b688-b859450a9b47","Type":"ContainerDied","Data":"a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97"}
Jul 24 16:30:12.023443 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:12.023426    2576 scope.go:117] "RemoveContainer" containerID="a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97"
Jul 24 16:30:12.024517 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:12.024488    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:30:12.024575 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:12.024554    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:30:12.024610 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:12.024585    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 24 16:30:13.030554 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.030517    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" event={"ID":"506dbf6e-1678-4e42-b688-b859450a9b47","Type":"ContainerStarted","Data":"20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01"}
Jul 24 16:30:13.283503 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283419    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"]
Jul 24 16:30:13.283821 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283807    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a677a159-699f-4f73-ae05-3738e47ea5a7" containerName="cleanup"
Jul 24 16:30:13.283864 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283824    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a677a159-699f-4f73-ae05-3738e47ea5a7" containerName="cleanup"
Jul 24 16:30:13.283864 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283840    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae09dfd-795f-4ea5-a50d-f8d6087d798e" containerName="cleanup"
Jul 24 16:30:13.283864 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283845    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae09dfd-795f-4ea5-a50d-f8d6087d798e" containerName="cleanup"
Jul 24 16:30:13.283864 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283851    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec79287b-3d47-4fc9-a71d-b7d988d91b2b" containerName="cleanup"
Jul 24 16:30:13.283864 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283856    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec79287b-3d47-4fc9-a71d-b7d988d91b2b" containerName="cleanup"
Jul 24 16:30:13.283864 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283864    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16571df9-8318-4691-a55f-089e5eaf786a" containerName="cleanup"
Jul 24 16:30:13.284048 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283869    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="16571df9-8318-4691-a55f-089e5eaf786a" containerName="cleanup"
Jul 24 16:30:13.284048 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283931    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="16571df9-8318-4691-a55f-089e5eaf786a" containerName="cleanup"
Jul 24 16:30:13.284048 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283938    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec79287b-3d47-4fc9-a71d-b7d988d91b2b" containerName="cleanup"
Jul 24 16:30:13.284048 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283947    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a677a159-699f-4f73-ae05-3738e47ea5a7" containerName="cleanup"
Jul 24 16:30:13.284048 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.283952    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae09dfd-795f-4ea5-a50d-f8d6087d798e" containerName="cleanup"
Jul 24 16:30:13.285967 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.285951    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"
Jul 24 16:30:13.288353 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.288326    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 16:30:13.295193 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.295163    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"]
Jul 24 16:30:13.452713 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.452668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6774f55-c0ed-45ec-b337-b4c2e1603bc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds\" (UID: \"d6774f55-c0ed-45ec-b337-b4c2e1603bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"
Jul 24 16:30:13.542188 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.542085    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:30:13.542417 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.542389    2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 24 16:30:13.542466 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.542445    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 24 16:30:13.554284 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.554226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6774f55-c0ed-45ec-b337-b4c2e1603bc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds\" (UID: \"d6774f55-c0ed-45ec-b337-b4c2e1603bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"
Jul 24 16:30:13.555066 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.555038    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6774f55-c0ed-45ec-b337-b4c2e1603bc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds\" (UID: \"d6774f55-c0ed-45ec-b337-b4c2e1603bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"
Jul 24 16:30:13.597685 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.597646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"
Jul 24 16:30:13.774614 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:13.772756    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"]
Jul 24 16:30:13.776394 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:30:13.776346    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6774f55_c0ed_45ec_b337_b4c2e1603bc3.slice/crio-999e5d1792c035a9c1016842486d87cb6325ceff816868d0b9e32dc8e46772ff WatchSource:0}: Error finding container 999e5d1792c035a9c1016842486d87cb6325ceff816868d0b9e32dc8e46772ff: Status 404 returned error can't find the container with id 999e5d1792c035a9c1016842486d87cb6325ceff816868d0b9e32dc8e46772ff
Jul 24 16:30:14.037589 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:14.037552    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds" event={"ID":"d6774f55-c0ed-45ec-b337-b4c2e1603bc3","Type":"ContainerStarted","Data":"999e5d1792c035a9c1016842486d87cb6325ceff816868d0b9e32dc8e46772ff"}
Jul 24 16:30:14.542171 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:14.542077    2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 24 16:30:14.542171 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:14.542135    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 24 16:30:15.044777 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:15.044745    2576 generic.go:356] "Generic (PLEG): container finished" podID="d6774f55-c0ed-45ec-b337-b4c2e1603bc3" containerID="0ec3d6d30211a855b087de11e8bf7a2531cf387ce205bf01381c29b5711f8bb3" exitCode=0
Jul 24 16:30:15.045209 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:15.044831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds" event={"ID":"d6774f55-c0ed-45ec-b337-b4c2e1603bc3","Type":"ContainerDied","Data":"0ec3d6d30211a855b087de11e8bf7a2531cf387ce205bf01381c29b5711f8bb3"}
Jul 24 16:30:15.542234 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:15.542194    2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 24 16:30:15.542462 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:15.542285    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 24 16:30:16.225029 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:16.225003    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"
Jul 24 16:30:16.294725 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:16.294690    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6774f55-c0ed-45ec-b337-b4c2e1603bc3-maas-api-service-ca\") pod \"d6774f55-c0ed-45ec-b337-b4c2e1603bc3\" (UID: \"d6774f55-c0ed-45ec-b337-b4c2e1603bc3\") "
Jul 24 16:30:16.295078 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:16.295054    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6774f55-c0ed-45ec-b337-b4c2e1603bc3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6774f55-c0ed-45ec-b337-b4c2e1603bc3" (UID: "d6774f55-c0ed-45ec-b337-b4c2e1603bc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:30:16.395520 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:16.395490    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6774f55-c0ed-45ec-b337-b4c2e1603bc3-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:16.542606 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:16.542571    2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 24 16:30:16.542761 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:16.542625    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 24 16:30:17.062171 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:17.062133    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds" event={"ID":"d6774f55-c0ed-45ec-b337-b4c2e1603bc3","Type":"ContainerDied","Data":"999e5d1792c035a9c1016842486d87cb6325ceff816868d0b9e32dc8e46772ff"}
Jul 24 16:30:17.062171 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:17.062154    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"
Jul 24 16:30:17.062387 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:17.062181    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="999e5d1792c035a9c1016842486d87cb6325ceff816868d0b9e32dc8e46772ff"
Jul 24 16:30:17.546182 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:17.546151    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:30:17.546607 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:17.546475    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:30:17.547277 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:17.547261    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:30:18.999978 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:18.999943    2576 scope.go:117] "RemoveContainer" containerID="f8b82adbf619ef72010339f85a283ef9b18397865f0977f79ded3cfa628daf28"
Jul 24 16:30:20.084509 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:20.084425    2576 generic.go:356] "Generic (PLEG): container finished" podID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerID="c9c109f7a3cb7cd51db2151eda1785647de214b53767b9a5bd1c7c5df80d9bff" exitCode=0
Jul 24 16:30:20.084898 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:20.084503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z" event={"ID":"4671a1f7-a63a-4cb8-95b6-d6a5e9911efa","Type":"ContainerDied","Data":"c9c109f7a3cb7cd51db2151eda1785647de214b53767b9a5bd1c7c5df80d9bff"}
Jul 24 16:30:20.084898 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:20.084567    2576 scope.go:117] "RemoveContainer" containerID="f8b82adbf619ef72010339f85a283ef9b18397865f0977f79ded3cfa628daf28"
Jul 24 16:30:21.109702 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:21.109675    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"]
Jul 24 16:30:21.231991 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:21.231965    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"
Jul 24 16:30:22.099410 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:22.099379    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"
Jul 24 16:30:22.099614 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:22.099378    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z" event={"ID":"4671a1f7-a63a-4cb8-95b6-d6a5e9911efa","Type":"ContainerDied","Data":"dd5537647006c7327d4954b4764dfcd35edd8581edb8b759293a3b842e9d800f"}
Jul 24 16:30:22.099614 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:22.099544    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd5537647006c7327d4954b4764dfcd35edd8581edb8b759293a3b842e9d800f"
Jul 24 16:30:22.135621 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:22.135582    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"]
Jul 24 16:30:22.139332 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:22.139305    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-4c4c5c-29748510-qfm6z"]
Jul 24 16:30:23.006084 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:23.006046    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" path="/var/lib/kubelet/pods/4671a1f7-a63a-4cb8-95b6-d6a5e9911efa/volumes"
Jul 24 16:30:28.270357 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:28.270319    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:30:28.379703 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:28.379666    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"]
Jul 24 16:30:28.382454 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:28.382426    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4c4c5c-npkds"]
Jul 24 16:30:29.005077 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:29.005040    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6774f55-c0ed-45ec-b337-b4c2e1603bc3" path="/var/lib/kubelet/pods/d6774f55-c0ed-45ec-b337-b4c2e1603bc3/volumes"
Jul 24 16:30:34.641377 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:34.641344    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"]
Jul 24 16:30:34.641883 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:34.641573    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy" containerID="cri-o://20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01" gracePeriod=30
Jul 24 16:30:38.316869 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.316827    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"]
Jul 24 16:30:38.317394 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317361    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerName="cleanup"
Jul 24 16:30:38.317394 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317380    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerName="cleanup"
Jul 24 16:30:38.317518 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317403    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6774f55-c0ed-45ec-b337-b4c2e1603bc3" containerName="revoke-keys"
Jul 24 16:30:38.317518 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317412    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6774f55-c0ed-45ec-b337-b4c2e1603bc3" containerName="revoke-keys"
Jul 24 16:30:38.317518 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317422    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerName="cleanup"
Jul 24 16:30:38.317518 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317430    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerName="cleanup"
Jul 24 16:30:38.317518 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317443    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerName="cleanup"
Jul 24 16:30:38.317518 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317452    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerName="cleanup"
Jul 24 16:30:38.317804 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317545    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6774f55-c0ed-45ec-b337-b4c2e1603bc3" containerName="revoke-keys"
Jul 24 16:30:38.317804 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317559    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerName="cleanup"
Jul 24 16:30:38.317804 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.317570    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerName="cleanup"
Jul 24 16:30:38.324275 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.323294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"
Jul 24 16:30:38.326835 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.326810    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\""
Jul 24 16:30:38.327103 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.326822    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 16:30:38.328837 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.328809    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"]
Jul 24 16:30:38.392021 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.391989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6be53f-7772-4b57-818d-408238ffecfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f\" (UID: \"0a6be53f-7772-4b57-818d-408238ffecfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"
Jul 24 16:30:38.492992 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.492958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6be53f-7772-4b57-818d-408238ffecfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f\" (UID: \"0a6be53f-7772-4b57-818d-408238ffecfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"
Jul 24 16:30:38.493690 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.493667    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6be53f-7772-4b57-818d-408238ffecfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f\" (UID: \"0a6be53f-7772-4b57-818d-408238ffecfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"
Jul 24 16:30:38.637165 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.637079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"
Jul 24 16:30:38.773347 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:38.773321    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"]
Jul 24 16:30:38.774872 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:30:38.774841    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6be53f_7772_4b57_818d_408238ffecfe.slice/crio-6c1f20d7a82b67bd21ec1f8e07325b3104937c2b45fc92a4a1e88cdc65d9049f WatchSource:0}: Error finding container 6c1f20d7a82b67bd21ec1f8e07325b3104937c2b45fc92a4a1e88cdc65d9049f: Status 404 returned error can't find the container with id 6c1f20d7a82b67bd21ec1f8e07325b3104937c2b45fc92a4a1e88cdc65d9049f
Jul 24 16:30:39.196036 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:39.196000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f" event={"ID":"0a6be53f-7772-4b57-818d-408238ffecfe","Type":"ContainerStarted","Data":"f143970a5f703b41512fed6b2792661067860f29efbec120bace6f46d1f03bc7"}
Jul 24 16:30:39.196205 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:39.196059    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f" event={"ID":"0a6be53f-7772-4b57-818d-408238ffecfe","Type":"ContainerStarted","Data":"6c1f20d7a82b67bd21ec1f8e07325b3104937c2b45fc92a4a1e88cdc65d9049f"}
Jul 24 16:30:39.212745 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:39.212640    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f" podStartSLOduration=1.212622907 podStartE2EDuration="1.212622907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:30:39.210809252 +0000 UTC m=+2998.695369849" watchObservedRunningTime="2026-07-24 16:30:39.212622907 +0000 UTC m=+2998.697183505"
Jul 24 16:30:39.904175 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:39.904112    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:30:40.009695 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.009662    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-envoy\") pod \"506dbf6e-1678-4e42-b688-b859450a9b47\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") "
Jul 24 16:30:40.009695 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.009704    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506dbf6e-1678-4e42-b688-b859450a9b47-istiod-ca-cert\") pod \"506dbf6e-1678-4e42-b688-b859450a9b47\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") "
Jul 24 16:30:40.009969 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.009741    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-data\") pod \"506dbf6e-1678-4e42-b688-b859450a9b47\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") "
Jul 24 16:30:40.009969 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.009756    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-istio-token\") pod \"506dbf6e-1678-4e42-b688-b859450a9b47\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") "
Jul 24 16:30:40.009969 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.009774    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-credential-socket\") pod \"506dbf6e-1678-4e42-b688-b859450a9b47\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") "
Jul 24 16:30:40.009969 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.009901    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdmsx\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-kube-api-access-sdmsx\") pod \"506dbf6e-1678-4e42-b688-b859450a9b47\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") "
Jul 24 16:30:40.009969 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.009962    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-certs\") pod \"506dbf6e-1678-4e42-b688-b859450a9b47\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") "
Jul 24 16:30:40.010230 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.010047    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506dbf6e-1678-4e42-b688-b859450a9b47-istio-podinfo\") pod \"506dbf6e-1678-4e42-b688-b859450a9b47\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") "
Jul 24 16:30:40.010230 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.010098    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-socket\") pod \"506dbf6e-1678-4e42-b688-b859450a9b47\" (UID: \"506dbf6e-1678-4e42-b688-b859450a9b47\") "
Jul 24 16:30:40.010368 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.010229    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "506dbf6e-1678-4e42-b688-b859450a9b47" (UID: "506dbf6e-1678-4e42-b688-b859450a9b47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:30:40.010368 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.010280    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506dbf6e-1678-4e42-b688-b859450a9b47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "506dbf6e-1678-4e42-b688-b859450a9b47" (UID: "506dbf6e-1678-4e42-b688-b859450a9b47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:30:40.010476 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.010458    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "506dbf6e-1678-4e42-b688-b859450a9b47" (UID: "506dbf6e-1678-4e42-b688-b859450a9b47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:30:40.010674 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.010483    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:40.010674 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.010506    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506dbf6e-1678-4e42-b688-b859450a9b47-istiod-ca-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:40.010674 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.010625    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-data" (OuterVolumeSpecName: "istio-data") pod "506dbf6e-1678-4e42-b688-b859450a9b47" (UID: "506dbf6e-1678-4e42-b688-b859450a9b47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:30:40.010869 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.010738    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "506dbf6e-1678-4e42-b688-b859450a9b47" (UID: "506dbf6e-1678-4e42-b688-b859450a9b47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:30:40.012313 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.012284    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "506dbf6e-1678-4e42-b688-b859450a9b47" (UID: "506dbf6e-1678-4e42-b688-b859450a9b47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:30:40.012432 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.012339    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-kube-api-access-sdmsx" (OuterVolumeSpecName: "kube-api-access-sdmsx") pod "506dbf6e-1678-4e42-b688-b859450a9b47" (UID: "506dbf6e-1678-4e42-b688-b859450a9b47"). InnerVolumeSpecName "kube-api-access-sdmsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:30:40.012432 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.012362    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-istio-token" (OuterVolumeSpecName: "istio-token") pod "506dbf6e-1678-4e42-b688-b859450a9b47" (UID: "506dbf6e-1678-4e42-b688-b859450a9b47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:30:40.012553 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.012505    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/506dbf6e-1678-4e42-b688-b859450a9b47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "506dbf6e-1678-4e42-b688-b859450a9b47" (UID: "506dbf6e-1678-4e42-b688-b859450a9b47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 16:30:40.111653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.111623    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-data\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:40.111653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.111649    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-istio-token\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:40.111653 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.111658    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-credential-socket\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:40.111888 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.111666    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdmsx\" (UniqueName: \"kubernetes.io/projected/506dbf6e-1678-4e42-b688-b859450a9b47-kube-api-access-sdmsx\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:40.111888 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.111676    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506dbf6e-1678-4e42-b688-b859450a9b47-istio-podinfo\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:40.111888 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.111686    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-workload-socket\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:40.111888 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.111694    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506dbf6e-1678-4e42-b688-b859450a9b47-istio-envoy\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:40.203349 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.203269    2576 generic.go:356] "Generic (PLEG): container finished" podID="506dbf6e-1678-4e42-b688-b859450a9b47" containerID="20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01" exitCode=0
Jul 24 16:30:40.203498 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.203348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" event={"ID":"506dbf6e-1678-4e42-b688-b859450a9b47","Type":"ContainerDied","Data":"20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01"}
Jul 24 16:30:40.203498 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.203383    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"
Jul 24 16:30:40.203498 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.203408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z" event={"ID":"506dbf6e-1678-4e42-b688-b859450a9b47","Type":"ContainerDied","Data":"c08f6d90a7cece93e7f43a48e4831117d49cefd19569eca8719f0912a1bd934f"}
Jul 24 16:30:40.203498 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.203436    2576 scope.go:117] "RemoveContainer" containerID="20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01"
Jul 24 16:30:40.204892 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.204866    2576 generic.go:356] "Generic (PLEG): container finished" podID="0a6be53f-7772-4b57-818d-408238ffecfe" containerID="f143970a5f703b41512fed6b2792661067860f29efbec120bace6f46d1f03bc7" exitCode=0
Jul 24 16:30:40.205183 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.204922    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f" event={"ID":"0a6be53f-7772-4b57-818d-408238ffecfe","Type":"ContainerDied","Data":"f143970a5f703b41512fed6b2792661067860f29efbec120bace6f46d1f03bc7"}
Jul 24 16:30:40.213193 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.213177    2576 scope.go:117] "RemoveContainer" containerID="a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97"
Jul 24 16:30:40.221823 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.221802    2576 scope.go:117] "RemoveContainer" containerID="20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01"
Jul 24 16:30:40.222072 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:30:40.222053    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01\": container with ID starting with 20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01 not found: ID does not exist" containerID="20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01"
Jul 24 16:30:40.222129 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.222082    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01"} err="failed to get container status \"20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01\": rpc error: code = NotFound desc = could not find container \"20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01\": container with ID starting with 20d8faac9520f796046269ab5149b07015cff50e86f5cf822cccd5f142adfc01 not found: ID does not exist"
Jul 24 16:30:40.222129 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.222102    2576 scope.go:117] "RemoveContainer" containerID="a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97"
Jul 24 16:30:40.222389 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:30:40.222368    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97\": container with ID starting with a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97 not found: ID does not exist" containerID="a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97"
Jul 24 16:30:40.222450 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.222397    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97"} err="failed to get container status \"a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97\": rpc error: code = NotFound desc = could not find container \"a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97\": container with ID starting with a004a3d55888a09e063ef10fb5d0cf9243a7739141a2bddba22628b3d2460e97 not found: ID does not exist"
Jul 24 16:30:40.237917 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.237890    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"]
Jul 24 16:30:40.245061 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:40.245039    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4c4c5c-openshift-default-5f9d49c8d-zcw2z"]
Jul 24 16:30:41.007585 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:41.007551    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" path="/var/lib/kubelet/pods/506dbf6e-1678-4e42-b688-b859450a9b47/volumes"
Jul 24 16:30:41.260512 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:41.260430    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:30:41.264983 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:41.264955    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:30:41.270853 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:41.270833    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:30:41.276520 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:41.276501    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:30:41.350485 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:41.350461    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"
Jul 24 16:30:41.423255 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:41.423217    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6be53f-7772-4b57-818d-408238ffecfe-maas-api-service-ca\") pod \"0a6be53f-7772-4b57-818d-408238ffecfe\" (UID: \"0a6be53f-7772-4b57-818d-408238ffecfe\") "
Jul 24 16:30:41.423518 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:41.423498    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6be53f-7772-4b57-818d-408238ffecfe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a6be53f-7772-4b57-818d-408238ffecfe" (UID: "0a6be53f-7772-4b57-818d-408238ffecfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:30:41.524856 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:41.524775    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6be53f-7772-4b57-818d-408238ffecfe-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:30:42.218075 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:42.218046    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"
Jul 24 16:30:42.218477 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:42.218076    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f" event={"ID":"0a6be53f-7772-4b57-818d-408238ffecfe","Type":"ContainerDied","Data":"6c1f20d7a82b67bd21ec1f8e07325b3104937c2b45fc92a4a1e88cdc65d9049f"}
Jul 24 16:30:42.218477 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:42.218128    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c1f20d7a82b67bd21ec1f8e07325b3104937c2b45fc92a4a1e88cdc65d9049f"
Jul 24 16:30:52.665256 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:52.665206    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:30:53.880688 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:53.880650    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk"]
Jul 24 16:30:53.885676 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:53.885650    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2c5dfe-29748510-vtwfk"]
Jul 24 16:30:55.005546 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:55.005513    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16571df9-8318-4691-a55f-089e5eaf786a" path="/var/lib/kubelet/pods/16571df9-8318-4691-a55f-089e5eaf786a/volumes"
Jul 24 16:30:58.403223 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:58.403188    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"]
Jul 24 16:30:58.405251 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:58.405219    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2c5dfe-qmx4f"]
Jul 24 16:30:59.005159 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:30:59.005126    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a6be53f-7772-4b57-818d-408238ffecfe" path="/var/lib/kubelet/pods/0a6be53f-7772-4b57-818d-408238ffecfe/volumes"
Jul 24 16:31:03.754342 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:03.754303    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"]
Jul 24 16:31:05.459664 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:05.459626    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:31:18.773767 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:18.773722    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw" podUID="386387b5-414d-4049-959d-0526224025d6" containerName="main" containerID="cri-o://0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6" gracePeriod=45
Jul 24 16:31:19.044849 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.044795    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:31:19.149022 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.148988    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-home\") pod \"386387b5-414d-4049-959d-0526224025d6\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") "
Jul 24 16:31:19.149206 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149037    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386387b5-414d-4049-959d-0526224025d6-tls-certs\") pod \"386387b5-414d-4049-959d-0526224025d6\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") "
Jul 24 16:31:19.149206 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149063    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-dshm\") pod \"386387b5-414d-4049-959d-0526224025d6\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") "
Jul 24 16:31:19.149206 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149090    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-tmp-dir\") pod \"386387b5-414d-4049-959d-0526224025d6\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") "
Jul 24 16:31:19.149206 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149116    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5zg8\" (UniqueName: \"kubernetes.io/projected/386387b5-414d-4049-959d-0526224025d6-kube-api-access-x5zg8\") pod \"386387b5-414d-4049-959d-0526224025d6\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") "
Jul 24 16:31:19.149206 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149145    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-model-cache\") pod \"386387b5-414d-4049-959d-0526224025d6\" (UID: \"386387b5-414d-4049-959d-0526224025d6\") "
Jul 24 16:31:19.149549 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149336    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-home" (OuterVolumeSpecName: "home") pod "386387b5-414d-4049-959d-0526224025d6" (UID: "386387b5-414d-4049-959d-0526224025d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:31:19.149549 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149454    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "386387b5-414d-4049-959d-0526224025d6" (UID: "386387b5-414d-4049-959d-0526224025d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:31:19.149549 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "386387b5-414d-4049-959d-0526224025d6" (UID: "386387b5-414d-4049-959d-0526224025d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:31:19.149682 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149629    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:31:19.149682 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149644    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:31:19.149682 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.149655    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:31:19.151429 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.151403    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-dshm" (OuterVolumeSpecName: "dshm") pod "386387b5-414d-4049-959d-0526224025d6" (UID: "386387b5-414d-4049-959d-0526224025d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:31:19.151844 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.151829    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386387b5-414d-4049-959d-0526224025d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "386387b5-414d-4049-959d-0526224025d6" (UID: "386387b5-414d-4049-959d-0526224025d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 16:31:19.151922 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.151862    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386387b5-414d-4049-959d-0526224025d6-kube-api-access-x5zg8" (OuterVolumeSpecName: "kube-api-access-x5zg8") pod "386387b5-414d-4049-959d-0526224025d6" (UID: "386387b5-414d-4049-959d-0526224025d6"). InnerVolumeSpecName "kube-api-access-x5zg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:31:19.250530 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.250500    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386387b5-414d-4049-959d-0526224025d6-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:31:19.250530 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.250526    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386387b5-414d-4049-959d-0526224025d6-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:31:19.250530 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.250536    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5zg8\" (UniqueName: \"kubernetes.io/projected/386387b5-414d-4049-959d-0526224025d6-kube-api-access-x5zg8\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:31:19.428202 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.428167    2576 generic.go:356] "Generic (PLEG): container finished" podID="386387b5-414d-4049-959d-0526224025d6" containerID="0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6" exitCode=0
Jul 24 16:31:19.428383 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.428212    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"
Jul 24 16:31:19.428383 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.428213    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw" event={"ID":"386387b5-414d-4049-959d-0526224025d6","Type":"ContainerDied","Data":"0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6"}
Jul 24 16:31:19.428383 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.428324    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw" event={"ID":"386387b5-414d-4049-959d-0526224025d6","Type":"ContainerDied","Data":"d17f4d6dd6e77b800052a82b344558e347f8d7a80bac82466a5c30e10509fcfe"}
Jul 24 16:31:19.428383 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.428348    2576 scope.go:117] "RemoveContainer" containerID="0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6"
Jul 24 16:31:19.438088 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.438072    2576 scope.go:117] "RemoveContainer" containerID="0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6"
Jul 24 16:31:19.438384 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:31:19.438356    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6\": container with ID starting with 0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6 not found: ID does not exist" containerID="0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6"
Jul 24 16:31:19.438442 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.438393    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6"} err="failed to get container status \"0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6\": rpc error: code = NotFound desc = could not find container \"0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6\": container with ID starting with 0e28779fbecc8da05abec87a3552c667c0620899101eae7c915bf6aaddf1faf6 not found: ID does not exist"
Jul 24 16:31:19.449267 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.449225    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"]
Jul 24 16:31:19.453390 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:19.453369    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2c5dfe/ipp-route-2c5dfe-kserve-56b848c9-wj8bw"]
Jul 24 16:31:21.007361 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:21.007330    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386387b5-414d-4049-959d-0526224025d6" path="/var/lib/kubelet/pods/386387b5-414d-4049-959d-0526224025d6/volumes"
Jul 24 16:31:46.922880 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.922847    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"]
Jul 24 16:31:46.923436 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923342    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy"
Jul 24 16:31:46.923436 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923360    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy"
Jul 24 16:31:46.923436 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923384    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386387b5-414d-4049-959d-0526224025d6" containerName="main"
Jul 24 16:31:46.923436 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923393    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="386387b5-414d-4049-959d-0526224025d6" containerName="main"
Jul 24 16:31:46.923436 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923405    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a6be53f-7772-4b57-818d-408238ffecfe" containerName="revoke-keys"
Jul 24 16:31:46.923436 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923414    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a6be53f-7772-4b57-818d-408238ffecfe" containerName="revoke-keys"
Jul 24 16:31:46.923756 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923514    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy"
Jul 24 16:31:46.923756 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923525    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4671a1f7-a63a-4cb8-95b6-d6a5e9911efa" containerName="cleanup"
Jul 24 16:31:46.923756 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923536    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="386387b5-414d-4049-959d-0526224025d6" containerName="main"
Jul 24 16:31:46.923756 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.923546    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a6be53f-7772-4b57-818d-408238ffecfe" containerName="revoke-keys"
Jul 24 16:31:46.928110 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.928090    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"
Jul 24 16:31:46.931358 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.931334    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 16:31:46.931686 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.931402    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\""
Jul 24 16:31:46.933479 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:46.933455    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"]
Jul 24 16:31:47.090339 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:47.090290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f79e65-3ecd-4a20-a017-e22649553bdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z\" (UID: \"02f79e65-3ecd-4a20-a017-e22649553bdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"
Jul 24 16:31:47.191269 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:47.191153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f79e65-3ecd-4a20-a017-e22649553bdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z\" (UID: \"02f79e65-3ecd-4a20-a017-e22649553bdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"
Jul 24 16:31:47.191769 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:47.191749    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f79e65-3ecd-4a20-a017-e22649553bdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z\" (UID: \"02f79e65-3ecd-4a20-a017-e22649553bdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"
Jul 24 16:31:47.239946 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:47.239907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"
Jul 24 16:31:47.385335 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:47.385300    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"]
Jul 24 16:31:47.387157 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:31:47.387128    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f79e65_3ecd_4a20_a017_e22649553bdc.slice/crio-5f3387cf9429a9158347ea6b4a5df2309f5bfb77882d36d03fd367e796fd43cd WatchSource:0}: Error finding container 5f3387cf9429a9158347ea6b4a5df2309f5bfb77882d36d03fd367e796fd43cd: Status 404 returned error can't find the container with id 5f3387cf9429a9158347ea6b4a5df2309f5bfb77882d36d03fd367e796fd43cd
Jul 24 16:31:47.581821 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:47.581781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z" event={"ID":"02f79e65-3ecd-4a20-a017-e22649553bdc","Type":"ContainerStarted","Data":"5f3387cf9429a9158347ea6b4a5df2309f5bfb77882d36d03fd367e796fd43cd"}
Jul 24 16:31:48.589308 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:48.589211    2576 generic.go:356] "Generic (PLEG): container finished" podID="02f79e65-3ecd-4a20-a017-e22649553bdc" containerID="7fca14631dd078717c34b906a19d76ba7a4f8a7031d3aad2fa08c388b34f0a74" exitCode=0
Jul 24 16:31:48.589698 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:48.589300    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z" event={"ID":"02f79e65-3ecd-4a20-a017-e22649553bdc","Type":"ContainerDied","Data":"7fca14631dd078717c34b906a19d76ba7a4f8a7031d3aad2fa08c388b34f0a74"}
Jul 24 16:31:49.729574 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:49.729551    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"
Jul 24 16:31:49.814235 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:49.814198    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f79e65-3ecd-4a20-a017-e22649553bdc-maas-api-service-ca\") pod \"02f79e65-3ecd-4a20-a017-e22649553bdc\" (UID: \"02f79e65-3ecd-4a20-a017-e22649553bdc\") "
Jul 24 16:31:49.814547 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:49.814524    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f79e65-3ecd-4a20-a017-e22649553bdc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02f79e65-3ecd-4a20-a017-e22649553bdc" (UID: "02f79e65-3ecd-4a20-a017-e22649553bdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:31:49.915908 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:49.915878    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f79e65-3ecd-4a20-a017-e22649553bdc-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:31:50.602581 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:50.602551    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"
Jul 24 16:31:50.602801 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:50.602599    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z" event={"ID":"02f79e65-3ecd-4a20-a017-e22649553bdc","Type":"ContainerDied","Data":"5f3387cf9429a9158347ea6b4a5df2309f5bfb77882d36d03fd367e796fd43cd"}
Jul 24 16:31:50.602801 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:50.602654    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f3387cf9429a9158347ea6b4a5df2309f5bfb77882d36d03fd367e796fd43cd"
Jul 24 16:31:57.710524 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:57.710479    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5"]
Jul 24 16:31:57.713133 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:57.713110    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c8631-29748510-qsqq5"]
Jul 24 16:31:59.005103 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:59.005066    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec79287b-3d47-4fc9-a71d-b7d988d91b2b" path="/var/lib/kubelet/pods/ec79287b-3d47-4fc9-a71d-b7d988d91b2b/volumes"
Jul 24 16:31:59.674201 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:31:59.674165    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:32:02.002394 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:02.002348    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"]
Jul 24 16:32:02.004572 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:02.004550    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c8631-2gt8z"]
Jul 24 16:32:03.006044 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:03.006006    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02f79e65-3ecd-4a20-a017-e22649553bdc" path="/var/lib/kubelet/pods/02f79e65-3ecd-4a20-a017-e22649553bdc/volumes"
Jul 24 16:32:08.412234 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.412205    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"]
Jul 24 16:32:08.412696 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.412470    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy" containerID="cri-o://55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557" gracePeriod=30
Jul 24 16:32:08.817198 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.817093    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"]
Jul 24 16:32:08.817477 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.817465    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy"
Jul 24 16:32:08.817523 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.817480    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy"
Jul 24 16:32:08.817523 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.817499    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f79e65-3ecd-4a20-a017-e22649553bdc" containerName="revoke-keys"
Jul 24 16:32:08.817523 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.817505    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f79e65-3ecd-4a20-a017-e22649553bdc" containerName="revoke-keys"
Jul 24 16:32:08.817616 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.817565    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="506dbf6e-1678-4e42-b688-b859450a9b47" containerName="istio-proxy"
Jul 24 16:32:08.817616 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.817576    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02f79e65-3ecd-4a20-a017-e22649553bdc" containerName="revoke-keys"
Jul 24 16:32:08.827164 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.827133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"
Jul 24 16:32:08.828886 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.828857    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"]
Jul 24 16:32:08.830155 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.830133    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 16:32:08.830295 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.830160    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\""
Jul 24 16:32:08.875879 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.875851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6531450-d792-41af-bd5a-ea9d32669706-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ed0643-k787q\" (UID: \"f6531450-d792-41af-bd5a-ea9d32669706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"
Jul 24 16:32:08.977139 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.977104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6531450-d792-41af-bd5a-ea9d32669706-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ed0643-k787q\" (UID: \"f6531450-d792-41af-bd5a-ea9d32669706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"
Jul 24 16:32:08.977710 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:08.977692    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6531450-d792-41af-bd5a-ea9d32669706-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ed0643-k787q\" (UID: \"f6531450-d792-41af-bd5a-ea9d32669706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"
Jul 24 16:32:09.138980 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:09.138905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"
Jul 24 16:32:09.274115 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:09.274091    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"]
Jul 24 16:32:09.275912 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:32:09.275883    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6531450_d792_41af_bd5a_ea9d32669706.slice/crio-ced2ba60f9231cc9ce7835040ce4cd7a766a710eb0b87b51bffc6fddb013c617 WatchSource:0}: Error finding container ced2ba60f9231cc9ce7835040ce4cd7a766a710eb0b87b51bffc6fddb013c617: Status 404 returned error can't find the container with id ced2ba60f9231cc9ce7835040ce4cd7a766a710eb0b87b51bffc6fddb013c617
Jul 24 16:32:09.713392 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:09.713360    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q" event={"ID":"f6531450-d792-41af-bd5a-ea9d32669706","Type":"ContainerStarted","Data":"908cfb82069cf7c1fea1890fcb0363eb3cefbf489954450ef232e447e89729eb"}
Jul 24 16:32:09.713764 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:09.713411    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q" event={"ID":"f6531450-d792-41af-bd5a-ea9d32669706","Type":"ContainerStarted","Data":"ced2ba60f9231cc9ce7835040ce4cd7a766a710eb0b87b51bffc6fddb013c617"}
Jul 24 16:32:09.727931 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:09.727841    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q" podStartSLOduration=1.727826423 podStartE2EDuration="1.727826423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:32:09.72685987 +0000 UTC m=+3089.211420468" watchObservedRunningTime="2026-07-24 16:32:09.727826423 +0000 UTC m=+3089.212387074"
Jul 24 16:32:10.720194 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:10.720160    2576 generic.go:356] "Generic (PLEG): container finished" podID="f6531450-d792-41af-bd5a-ea9d32669706" containerID="908cfb82069cf7c1fea1890fcb0363eb3cefbf489954450ef232e447e89729eb" exitCode=0
Jul 24 16:32:10.720671 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:10.720259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q" event={"ID":"f6531450-d792-41af-bd5a-ea9d32669706","Type":"ContainerDied","Data":"908cfb82069cf7c1fea1890fcb0363eb3cefbf489954450ef232e447e89729eb"}
Jul 24 16:32:11.857953 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:11.857931    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"
Jul 24 16:32:11.901926 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:11.901900    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6531450-d792-41af-bd5a-ea9d32669706-maas-api-service-ca\") pod \"f6531450-d792-41af-bd5a-ea9d32669706\" (UID: \"f6531450-d792-41af-bd5a-ea9d32669706\") "
Jul 24 16:32:11.902236 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:11.902215    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6531450-d792-41af-bd5a-ea9d32669706-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6531450-d792-41af-bd5a-ea9d32669706" (UID: "f6531450-d792-41af-bd5a-ea9d32669706"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:32:12.002869 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:12.002799    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6531450-d792-41af-bd5a-ea9d32669706-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:12.732702 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:12.732667    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"
Jul 24 16:32:12.732988 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:12.732710    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q" event={"ID":"f6531450-d792-41af-bd5a-ea9d32669706","Type":"ContainerDied","Data":"ced2ba60f9231cc9ce7835040ce4cd7a766a710eb0b87b51bffc6fddb013c617"}
Jul 24 16:32:12.732988 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:12.732801    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ced2ba60f9231cc9ce7835040ce4cd7a766a710eb0b87b51bffc6fddb013c617"
Jul 24 16:32:13.667919 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.667887    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:32:13.718423 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718359    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-envoy\") pod \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") "
Jul 24 16:32:13.718423 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718406    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-certs\") pod \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") "
Jul 24 16:32:13.718590 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718451    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-socket\") pod \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") "
Jul 24 16:32:13.718590 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718478    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-token\") pod \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") "
Jul 24 16:32:13.718590 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718492    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-credential-socket\") pod \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") "
Jul 24 16:32:13.718590 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718537    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9cpr\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-kube-api-access-b9cpr\") pod \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") "
Jul 24 16:32:13.718590 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718565    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-data\") pod \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") "
Jul 24 16:32:13.718590 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718590    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-podinfo\") pod \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") "
Jul 24 16:32:13.718898 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718656    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istiod-ca-cert\") pod \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\" (UID: \"5b8fb632-f838-43d2-9b0f-85c4d07acc77\") "
Jul 24 16:32:13.718898 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718703    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b8fb632-f838-43d2-9b0f-85c4d07acc77" (UID: "5b8fb632-f838-43d2-9b0f-85c4d07acc77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:32:13.718898 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718866    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b8fb632-f838-43d2-9b0f-85c4d07acc77" (UID: "5b8fb632-f838-43d2-9b0f-85c4d07acc77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:32:13.719066 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718980    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-socket\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:13.719066 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.718999    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-credential-socket\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:13.719208 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.719188    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b8fb632-f838-43d2-9b0f-85c4d07acc77" (UID: "5b8fb632-f838-43d2-9b0f-85c4d07acc77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 16:32:13.719506 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.719481    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b8fb632-f838-43d2-9b0f-85c4d07acc77" (UID: "5b8fb632-f838-43d2-9b0f-85c4d07acc77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:32:13.719885 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.719857    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b8fb632-f838-43d2-9b0f-85c4d07acc77" (UID: "5b8fb632-f838-43d2-9b0f-85c4d07acc77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:32:13.720778 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.720752    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b8fb632-f838-43d2-9b0f-85c4d07acc77" (UID: "5b8fb632-f838-43d2-9b0f-85c4d07acc77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:32:13.720930 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.720908    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-kube-api-access-b9cpr" (OuterVolumeSpecName: "kube-api-access-b9cpr") pod "5b8fb632-f838-43d2-9b0f-85c4d07acc77" (UID: "5b8fb632-f838-43d2-9b0f-85c4d07acc77"). InnerVolumeSpecName "kube-api-access-b9cpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 16:32:13.721216 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.721193    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b8fb632-f838-43d2-9b0f-85c4d07acc77" (UID: "5b8fb632-f838-43d2-9b0f-85c4d07acc77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 16:32:13.721330 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.721213    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b8fb632-f838-43d2-9b0f-85c4d07acc77" (UID: "5b8fb632-f838-43d2-9b0f-85c4d07acc77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 16:32:13.739462 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.739430    2576 generic.go:356] "Generic (PLEG): container finished" podID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerID="55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557" exitCode=0
Jul 24 16:32:13.739573 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.739479    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"
Jul 24 16:32:13.739573 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.739504    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" event={"ID":"5b8fb632-f838-43d2-9b0f-85c4d07acc77","Type":"ContainerDied","Data":"55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557"}
Jul 24 16:32:13.739573 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.739561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq" event={"ID":"5b8fb632-f838-43d2-9b0f-85c4d07acc77","Type":"ContainerDied","Data":"786dd5a4f2b51ccdcddb4c94c75e2e5cc81f8219fdc901b25742fc7747739159"}
Jul 24 16:32:13.739756 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.739584    2576 scope.go:117] "RemoveContainer" containerID="55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557"
Jul 24 16:32:13.751448 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.751424    2576 scope.go:117] "RemoveContainer" containerID="887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0"
Jul 24 16:32:13.761359 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.761342    2576 scope.go:117] "RemoveContainer" containerID="55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557"
Jul 24 16:32:13.761595 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:32:13.761577    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557\": container with ID starting with 55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557 not found: ID does not exist" containerID="55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557"
Jul 24 16:32:13.761649 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.761602    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557"} err="failed to get container status \"55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557\": rpc error: code = NotFound desc = could not find container \"55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557\": container with ID starting with 55a8f3c348c6bbc9cace351f6e834453beb17b3e4e34014d3a7a2e1685fb0557 not found: ID does not exist"
Jul 24 16:32:13.761649 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.761616    2576 scope.go:117] "RemoveContainer" containerID="887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0"
Jul 24 16:32:13.761866 ip-10-0-131-210 kubenswrapper[2576]: E0724 16:32:13.761847    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0\": container with ID starting with 887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0 not found: ID does not exist" containerID="887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0"
Jul 24 16:32:13.761912 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.761871    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0"} err="failed to get container status \"887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0\": rpc error: code = NotFound desc = could not find container \"887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0\": container with ID starting with 887d8bdd85b3ee88f02ce2c03997e2845187ffe735d1e3054c2bee36adebaba0 not found: ID does not exist"
Jul 24 16:32:13.770564 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.770541    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"]
Jul 24 16:32:13.772637 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.772613    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c8631-openshift-default-6c7f474fc8-qklnq"]
Jul 24 16:32:13.819680 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.819656    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-token\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:13.819680 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.819677    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9cpr\" (UniqueName: \"kubernetes.io/projected/5b8fb632-f838-43d2-9b0f-85c4d07acc77-kube-api-access-b9cpr\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:13.819800 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.819687    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-data\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:13.819800 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.819695    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-podinfo\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:13.819800 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.819703    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istiod-ca-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:13.819800 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.819711    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-istio-envoy\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:13.819800 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:13.819718    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8fb632-f838-43d2-9b0f-85c4d07acc77-workload-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 24 16:32:15.006399 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:15.006364    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" path="/var/lib/kubelet/pods/5b8fb632-f838-43d2-9b0f-85c4d07acc77/volumes"
Jul 24 16:32:22.461047 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:22.461015    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:32:23.892608 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:23.892536    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"]
Jul 24 16:32:23.895323 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:23.895298    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed0643-k787q"]
Jul 24 16:32:25.005754 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:25.005720    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6531450-d792-41af-bd5a-ea9d32669706" path="/var/lib/kubelet/pods/f6531450-d792-41af-bd5a-ea9d32669706/volumes"
Jul 24 16:32:41.403803 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:41.403760    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb"]
Jul 24 16:32:41.405947 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:41.405919    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5aebfb-29748510-shbmb"]
Jul 24 16:32:43.004764 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:43.004727    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae09dfd-795f-4ea5-a50d-f8d6087d798e" path="/var/lib/kubelet/pods/7ae09dfd-795f-4ea5-a50d-f8d6087d798e/volumes"
Jul 24 16:32:44.365211 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:44.365181    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-42j5d"]
Jul 24 16:32:53.168231 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:53.168196    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d58695fd4-kjl7p_aec2c1d1-6069-4021-a393-e94a9b116163/authorino/0.log"
Jul 24 16:32:59.276866 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:59.276833    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-crbzt_da4d0b72-d67a-4eee-b14c-2a18465da204/server/0.log"
Jul 24 16:32:59.401558 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:59.401525    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-52rtw_f0abdb77-eff9-4352-b303-9cc347bef506/manager/0.log"
Jul 24 16:32:59.517459 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:59.517437    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6798866cc5-8zmjb_b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab/copy-manifests/0.log"
Jul 24 16:32:59.524395 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:32:59.524370    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6798866cc5-8zmjb_b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab/manager/0.log"
Jul 24 16:33:00.464544 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:00.464516    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748510-9whcz_a677a159-699f-4f73-ae05-3738e47ea5a7/cleanup/0.log"
Jul 24 16:33:01.379686 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.379657    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh_4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a/util/0.log"
Jul 24 16:33:01.388049 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.388027    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh_4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a/pull/0.log"
Jul 24 16:33:01.396265 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.396223    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh_4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a/extract/0.log"
Jul 24 16:33:01.516424 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.516394    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc_27d00aee-53a1-4b28-a551-3277d409dc5e/util/0.log"
Jul 24 16:33:01.523877 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.523850    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc_27d00aee-53a1-4b28-a551-3277d409dc5e/pull/0.log"
Jul 24 16:33:01.531323 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.531302    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc_27d00aee-53a1-4b28-a551-3277d409dc5e/extract/0.log"
Jul 24 16:33:01.640507 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.640436    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg_9d3e023d-c365-48a3-a523-5f34db6c3993/util/0.log"
Jul 24 16:33:01.647442 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.647419    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg_9d3e023d-c365-48a3-a523-5f34db6c3993/pull/0.log"
Jul 24 16:33:01.654993 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.654972    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg_9d3e023d-c365-48a3-a523-5f34db6c3993/extract/0.log"
Jul 24 16:33:01.771870 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.771849    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn_6e4b692e-fd8f-47c6-a5eb-8734b011753a/pull/0.log"
Jul 24 16:33:01.778367 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.778339    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn_6e4b692e-fd8f-47c6-a5eb-8734b011753a/extract/0.log"
Jul 24 16:33:01.784720 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.784703    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn_6e4b692e-fd8f-47c6-a5eb-8734b011753a/util/0.log"
Jul 24 16:33:01.903841 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:01.903809    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d58695fd4-kjl7p_aec2c1d1-6069-4021-a393-e94a9b116163/authorino/0.log"
Jul 24 16:33:02.279566 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:02.279450    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-s7rmt_8ed42646-07f6-400f-92b5-5067c236e340/kuadrant-console-plugin/0.log"
Jul 24 16:33:02.653900 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:02.653870    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-42j5d_3a04fd46-d9fe-46e0-b2cb-033a886a55c2/limitador/0.log"
Jul 24 16:33:03.495700 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:03.495660    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5688cff95c-qjz2g_96fa6894-d9b9-4531-9bd4-a2b3a01c262f/kube-auth-proxy/0.log"
Jul 24 16:33:03.975519 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:03.975477    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798f8fd9f9-zr8tr_8814c4dc-52a8-4447-a645-adecc1b46d89/router/0.log"
Jul 24 16:33:04.899387 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:04.899351    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-2f4l4_53206d9b-ec6b-474d-9301-84a0e354fda3/main/0.log"
Jul 24 16:33:10.914689 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.914656    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7s9sw/must-gather-5xr65"]
Jul 24 16:33:10.915063 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.915001    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy"
Jul 24 16:33:10.915063 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.915013    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy"
Jul 24 16:33:10.915063 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.915024    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6531450-d792-41af-bd5a-ea9d32669706" containerName="revoke-keys"
Jul 24 16:33:10.915063 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.915029    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6531450-d792-41af-bd5a-ea9d32669706" containerName="revoke-keys"
Jul 24 16:33:10.915196 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.915092    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6531450-d792-41af-bd5a-ea9d32669706" containerName="revoke-keys"
Jul 24 16:33:10.915196 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.915101    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy"
Jul 24 16:33:10.915196 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.915107    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy"
Jul 24 16:33:10.915196 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.915166    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy"
Jul 24 16:33:10.915196 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.915171    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8fb632-f838-43d2-9b0f-85c4d07acc77" containerName="istio-proxy"
Jul 24 16:33:10.918499 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.918474    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s9sw/must-gather-5xr65"
Jul 24 16:33:10.921074 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.921049    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7s9sw\"/\"openshift-service-ca.crt\""
Jul 24 16:33:10.921221 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.921073    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7s9sw\"/\"default-dockercfg-fh7j9\""
Jul 24 16:33:10.922320 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.922254    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7s9sw\"/\"kube-root-ca.crt\""
Jul 24 16:33:10.929555 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:10.929534    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s9sw/must-gather-5xr65"]
Jul 24 16:33:11.005899 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:11.005872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0838760b-82eb-4b30-8aab-d3bd31aa157c-must-gather-output\") pod \"must-gather-5xr65\" (UID: \"0838760b-82eb-4b30-8aab-d3bd31aa157c\") " pod="openshift-must-gather-7s9sw/must-gather-5xr65"
Jul 24 16:33:11.006077 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:11.005932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4vn\" (UniqueName: \"kubernetes.io/projected/0838760b-82eb-4b30-8aab-d3bd31aa157c-kube-api-access-pl4vn\") pod \"must-gather-5xr65\" (UID: \"0838760b-82eb-4b30-8aab-d3bd31aa157c\") " pod="openshift-must-gather-7s9sw/must-gather-5xr65"
Jul 24 16:33:11.107063 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:11.107030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0838760b-82eb-4b30-8aab-d3bd31aa157c-must-gather-output\") pod \"must-gather-5xr65\" (UID: \"0838760b-82eb-4b30-8aab-d3bd31aa157c\") " pod="openshift-must-gather-7s9sw/must-gather-5xr65"
Jul 24 16:33:11.107260 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:11.107119    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4vn\" (UniqueName: \"kubernetes.io/projected/0838760b-82eb-4b30-8aab-d3bd31aa157c-kube-api-access-pl4vn\") pod \"must-gather-5xr65\" (UID: \"0838760b-82eb-4b30-8aab-d3bd31aa157c\") " pod="openshift-must-gather-7s9sw/must-gather-5xr65"
Jul 24 16:33:11.107365 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:11.107345    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0838760b-82eb-4b30-8aab-d3bd31aa157c-must-gather-output\") pod \"must-gather-5xr65\" (UID: \"0838760b-82eb-4b30-8aab-d3bd31aa157c\") " pod="openshift-must-gather-7s9sw/must-gather-5xr65"
Jul 24 16:33:11.120967 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:11.120938    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4vn\" (UniqueName: \"kubernetes.io/projected/0838760b-82eb-4b30-8aab-d3bd31aa157c-kube-api-access-pl4vn\") pod \"must-gather-5xr65\" (UID: \"0838760b-82eb-4b30-8aab-d3bd31aa157c\") " pod="openshift-must-gather-7s9sw/must-gather-5xr65"
Jul 24 16:33:11.230374 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:11.230304    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s9sw/must-gather-5xr65"
Jul 24 16:33:11.370329 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:11.370304    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s9sw/must-gather-5xr65"]
Jul 24 16:33:11.373040 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:33:11.373017    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0838760b_82eb_4b30_8aab_d3bd31aa157c.slice/crio-42759140da943e3995fc165962b0a873e98056d39934f18a596f97f5986cb2cb WatchSource:0}: Error finding container 42759140da943e3995fc165962b0a873e98056d39934f18a596f97f5986cb2cb: Status 404 returned error can't find the container with id 42759140da943e3995fc165962b0a873e98056d39934f18a596f97f5986cb2cb
Jul 24 16:33:11.375445 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:11.375427    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 16:33:12.056862 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:12.056822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s9sw/must-gather-5xr65" event={"ID":"0838760b-82eb-4b30-8aab-d3bd31aa157c","Type":"ContainerStarted","Data":"42759140da943e3995fc165962b0a873e98056d39934f18a596f97f5986cb2cb"}
Jul 24 16:33:13.067828 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:13.067782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s9sw/must-gather-5xr65" event={"ID":"0838760b-82eb-4b30-8aab-d3bd31aa157c","Type":"ContainerStarted","Data":"5ac1a6ee094b130868758214d69f42f7c7a34c1e577c2af9e9571017a6c827de"}
Jul 24 16:33:13.068282 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:13.067845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s9sw/must-gather-5xr65" event={"ID":"0838760b-82eb-4b30-8aab-d3bd31aa157c","Type":"ContainerStarted","Data":"e11524e31222de2e1e5abdcc628c30c2cc1551fe7a6ae282c5b927654dce1961"}
Jul 24 16:33:13.087374 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:13.087324    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7s9sw/must-gather-5xr65" podStartSLOduration=2.327722818 podStartE2EDuration="3.087309187s" totalImagesPullingTime="759.586369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:33:13.086367833 +0000 UTC m=+3152.570928431" watchObservedRunningTime="2026-07-24 16:33:13.087309187 +0000 UTC m=+3152.571869784"
Jul 24 16:33:13.720214 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:13.720176    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9k7ms_33d7b0af-08cf-4fa6-9f39-e461402d71f1/global-pull-secret-syncer/0.log"
Jul 24 16:33:13.815815 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:13.815783    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rfc4r_d3a39934-0ab7-466e-a958-7ffebde7bb5a/konnectivity-agent/0.log"
Jul 24 16:33:13.869508 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:13.869475    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-210.ec2.internal_6924941272464cf155cde721c0cf91a1/haproxy/0.log"
Jul 24 16:33:17.859089 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:17.859057    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh_4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a/extract/0.log"
Jul 24 16:33:17.887465 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:17.887434    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh_4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a/util/0.log"
Jul 24 16:33:17.910809 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:17.910777    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594gtgh_4cf4c4e6-6171-434d-8b2a-d0a7fe36f18a/pull/0.log"
Jul 24 16:33:17.958736 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:17.958702    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc_27d00aee-53a1-4b28-a551-3277d409dc5e/extract/0.log"
Jul 24 16:33:17.983672 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:17.983644    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc_27d00aee-53a1-4b28-a551-3277d409dc5e/util/0.log"
Jul 24 16:33:18.004759 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:18.004727    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bq4dc_27d00aee-53a1-4b28-a551-3277d409dc5e/pull/0.log"
Jul 24 16:33:18.033623 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:18.033597    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg_9d3e023d-c365-48a3-a523-5f34db6c3993/extract/0.log"
Jul 24 16:33:18.055563 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:18.055522    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg_9d3e023d-c365-48a3-a523-5f34db6c3993/util/0.log"
Jul 24 16:33:18.083463 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:18.083437    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73v88lg_9d3e023d-c365-48a3-a523-5f34db6c3993/pull/0.log"
Jul 24 16:33:18.131814 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:18.131740    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn_6e4b692e-fd8f-47c6-a5eb-8734b011753a/extract/0.log"
Jul 24 16:33:18.155273 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:18.155229    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn_6e4b692e-fd8f-47c6-a5eb-8734b011753a/util/0.log"
Jul 24 16:33:18.176081 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:18.176052    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dcpmn_6e4b692e-fd8f-47c6-a5eb-8734b011753a/pull/0.log"
Jul 24 16:33:18.716404 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:18.716319    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d58695fd4-kjl7p_aec2c1d1-6069-4021-a393-e94a9b116163/authorino/0.log"
Jul 24 16:33:18.804017 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:18.803984    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-s7rmt_8ed42646-07f6-400f-92b5-5067c236e340/kuadrant-console-plugin/0.log"
Jul 24 16:33:19.040099 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:19.040028    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-42j5d_3a04fd46-d9fe-46e0-b2cb-033a886a55c2/limitador/0.log"
Jul 24 16:33:21.025909 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:21.025874    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-r9c4b_147ab95a-ccc0-46b6-88f6-553b434f3268/cluster-monitoring-operator/0.log"
Jul 24 16:33:21.328870 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:21.328836    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpkwd_92032438-e0c7-4d52-836d-236573ff3419/node-exporter/0.log"
Jul 24 16:33:21.351405 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:21.351376    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpkwd_92032438-e0c7-4d52-836d-236573ff3419/kube-rbac-proxy/0.log"
Jul 24 16:33:21.373780 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:21.373752    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpkwd_92032438-e0c7-4d52-836d-236573ff3419/init-textfile/0.log"
Jul 24 16:33:23.140398 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.140367    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-6vwr4_e25ae61d-8fd4-499c-9d04-a58302f9886a/networking-console-plugin/0.log"
Jul 24 16:33:23.729453 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.729426    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/2.log"
Jul 24 16:33:23.737944 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.737913    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-9b8th_b9580587-864e-44e1-8d91-33cc9aa83cc5/console-operator/3.log"
Jul 24 16:33:23.798777 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.798740    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"]
Jul 24 16:33:23.806549 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.806514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:23.809805 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.809775    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"]
Jul 24 16:33:23.934396 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.934354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58nv\" (UniqueName: \"kubernetes.io/projected/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-kube-api-access-q58nv\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:23.934585 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.934412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-lib-modules\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:23.934585 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.934439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-podres\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:23.934585 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.934532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-sys\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:23.934585 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:23.934576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-proc\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.036191 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.036093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58nv\" (UniqueName: \"kubernetes.io/projected/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-kube-api-access-q58nv\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.036191 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.036132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-lib-modules\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.036191 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.036160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-podres\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.036495 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.036200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-sys\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.036495 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.036225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-proc\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.036495 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.036330    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-proc\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.036495 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.036337    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-lib-modules\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.036495 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.036411    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-sys\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.036495 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.036426    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-podres\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.044693 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.044664    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58nv\" (UniqueName: \"kubernetes.io/projected/5afd2698-d7b6-41d5-9be7-0ec0c7efc874-kube-api-access-q58nv\") pod \"perf-node-gather-daemonset-h8pzt\" (UID: \"5afd2698-d7b6-41d5-9be7-0ec0c7efc874\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.122601 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.122358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:24.319564 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:24.319535    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"]
Jul 24 16:33:24.322355 ip-10-0-131-210 kubenswrapper[2576]: W0724 16:33:24.322311    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5afd2698_d7b6_41d5_9be7_0ec0c7efc874.slice/crio-944a1a41d7c41b1192895e636705cdfa107d403b0ec4e96358728dad175255a7 WatchSource:0}: Error finding container 944a1a41d7c41b1192895e636705cdfa107d403b0ec4e96358728dad175255a7: Status 404 returned error can't find the container with id 944a1a41d7c41b1192895e636705cdfa107d403b0ec4e96358728dad175255a7
Jul 24 16:33:25.034544 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:25.034521    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-d99lr_c2d5ba91-0564-4242-aae2-1239c9666af1/volume-data-source-validator/0.log"
Jul 24 16:33:25.178994 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:25.178947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt" event={"ID":"5afd2698-d7b6-41d5-9be7-0ec0c7efc874","Type":"ContainerStarted","Data":"bfa981242b3ef49e37653a00ebdac8d2b9f7b81f49ce44be712662eb8fe26874"}
Jul 24 16:33:25.179288 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:25.179265    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt" event={"ID":"5afd2698-d7b6-41d5-9be7-0ec0c7efc874","Type":"ContainerStarted","Data":"944a1a41d7c41b1192895e636705cdfa107d403b0ec4e96358728dad175255a7"}
Jul 24 16:33:25.179476 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:25.179462    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:25.196664 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:25.196614    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt" podStartSLOduration=2.196596172 podStartE2EDuration="2.196596172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:33:25.194761579 +0000 UTC m=+3164.679322177" watchObservedRunningTime="2026-07-24 16:33:25.196596172 +0000 UTC m=+3164.681156770"
Jul 24 16:33:26.118381 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:26.118342    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tf88h_66b549b1-da81-4293-95c9-a7524f30bc40/dns/0.log"
Jul 24 16:33:26.138487 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:26.138451    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tf88h_66b549b1-da81-4293-95c9-a7524f30bc40/kube-rbac-proxy/0.log"
Jul 24 16:33:26.213235 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:26.213179    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgs2m_642babd4-24ba-47d0-899b-e8345129d52a/dns-node-resolver/0.log"
Jul 24 16:33:26.761476 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:26.761435    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vf9gg_f14cf9e9-32cf-4ce9-9761-16bed40d8a5e/node-ca/0.log"
Jul 24 16:33:27.867677 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:27.867635    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5688cff95c-qjz2g_96fa6894-d9b9-4531-9bd4-a2b3a01c262f/kube-auth-proxy/0.log"
Jul 24 16:33:28.008658 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:28.008617    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798f8fd9f9-zr8tr_8814c4dc-52a8-4447-a645-adecc1b46d89/router/0.log"
Jul 24 16:33:28.817945 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:28.817913    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x5l4p_76a699aa-4df7-4e65-bc5b-a87ab2c316e4/serve-healthcheck-canary/0.log"
Jul 24 16:33:29.269623 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:29.269587    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-rk57t_c2124fe0-2969-4497-862e-7ea34b7e487c/insights-operator/0.log"
Jul 24 16:33:29.273690 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:29.273667    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-rk57t_c2124fe0-2969-4497-862e-7ea34b7e487c/insights-operator/1.log"
Jul 24 16:33:29.417817 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:29.417785    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lx822_2d245c2e-8405-4b45-927b-cd0bcfbd5883/kube-rbac-proxy/0.log"
Jul 24 16:33:29.436788 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:29.436761    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lx822_2d245c2e-8405-4b45-927b-cd0bcfbd5883/exporter/0.log"
Jul 24 16:33:29.456458 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:29.456432    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lx822_2d245c2e-8405-4b45-927b-cd0bcfbd5883/extractor/0.log"
Jul 24 16:33:31.856163 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:31.856133    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-crbzt_da4d0b72-d67a-4eee-b14c-2a18465da204/server/0.log"
Jul 24 16:33:31.988431 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:31.988395    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-52rtw_f0abdb77-eff9-4352-b303-9cc347bef506/manager/0.log"
Jul 24 16:33:32.012944 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:32.012909    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6798866cc5-8zmjb_b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab/manager/0.log"
Jul 24 16:33:32.033897 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:32.033861    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6798866cc5-8zmjb_b8f8eb07-ec2a-4cca-9bda-7ecdb1f09fab/copy-manifests/0.log"
Jul 24 16:33:32.207738 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:32.206159    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-h8pzt"
Jul 24 16:33:33.309717 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:33.309670    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65dbfbfd5-mg884_68141270-4a21-4d80-84e7-5d0be3959fda/manager/0.log"
Jul 24 16:33:34.170753 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:34.170720    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5688cff95c-qjz2g_96fa6894-d9b9-4531-9bd4-a2b3a01c262f/kube-auth-proxy/0.log"
Jul 24 16:33:34.281214 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:34.281177    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798f8fd9f9-zr8tr_8814c4dc-52a8-4447-a645-adecc1b46d89/router/0.log"
Jul 24 16:33:35.911787 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:35.911746    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5688cff95c-qjz2g_96fa6894-d9b9-4531-9bd4-a2b3a01c262f/kube-auth-proxy/0.log"
Jul 24 16:33:36.025926 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:36.025900    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798f8fd9f9-zr8tr_8814c4dc-52a8-4447-a645-adecc1b46d89/router/0.log"
Jul 24 16:33:38.850756 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:38.850723    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7_a9c1f7cf-fbd5-4107-903d-a6538f15c309/kube-storage-version-migrator-operator/1.log"
Jul 24 16:33:38.852422 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:38.852396    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-dlxx7_a9c1f7cf-fbd5-4107-903d-a6538f15c309/kube-storage-version-migrator-operator/0.log"
Jul 24 16:33:40.102620 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.102594    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wk55j_0ad7bbb9-a8d6-40d9-a962-3ce602311838/kube-multus-additional-cni-plugins/0.log"
Jul 24 16:33:40.136654 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.136628    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wk55j_0ad7bbb9-a8d6-40d9-a962-3ce602311838/egress-router-binary-copy/0.log"
Jul 24 16:33:40.174535 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.174507    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wk55j_0ad7bbb9-a8d6-40d9-a962-3ce602311838/cni-plugins/0.log"
Jul 24 16:33:40.210423 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.210399    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wk55j_0ad7bbb9-a8d6-40d9-a962-3ce602311838/bond-cni-plugin/0.log"
Jul 24 16:33:40.245378 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.245347    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wk55j_0ad7bbb9-a8d6-40d9-a962-3ce602311838/routeoverride-cni/0.log"
Jul 24 16:33:40.270864 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.270839    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wk55j_0ad7bbb9-a8d6-40d9-a962-3ce602311838/whereabouts-cni-bincopy/0.log"
Jul 24 16:33:40.300432 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.300407    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wk55j_0ad7bbb9-a8d6-40d9-a962-3ce602311838/whereabouts-cni/0.log"
Jul 24 16:33:40.394084 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.394050    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tdm2k_de658cfe-d206-4123-9309-e2fe7309d4e5/kube-multus/0.log"
Jul 24 16:33:40.427938 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.427909    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2wl2t_d71a09df-224a-475a-9c69-4fe776c7cdb3/network-metrics-daemon/0.log"
Jul 24 16:33:40.459707 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:40.459664    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2wl2t_d71a09df-224a-475a-9c69-4fe776c7cdb3/kube-rbac-proxy/0.log"
Jul 24 16:33:41.754061 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:41.753970    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-controller/0.log"
Jul 24 16:33:41.783074 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:41.783035    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/0.log"
Jul 24 16:33:41.783074 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:41.783047    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovn-acl-logging/1.log"
Jul 24 16:33:41.814872 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:41.814846    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/kube-rbac-proxy-node/0.log"
Jul 24 16:33:41.850805 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:41.850778    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/kube-rbac-proxy-ovn-metrics/0.log"
Jul 24 16:33:41.882047 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:41.882021    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/northd/0.log"
Jul 24 16:33:41.923401 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:41.923377    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/nbdb/0.log"
Jul 24 16:33:41.956383 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:41.956353    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/sbdb/0.log"
Jul 24 16:33:42.080729 ip-10-0-131-210 kubenswrapper[2576]: I0724 16:33:42.080650    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf4lp_5a167781-23cd-4fec-880a-76ccee6874c5/ovnkube-controller/0.log"