Jul 23 18:16:44.024576 ip-10-0-140-91 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.408185    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415218    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415238    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415242    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415245    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415248    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415251    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415254    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 18:16:44.458857 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415256    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415259    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415262    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415265    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415267    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415270    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415273    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415275    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415278    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415292    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415295    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415299    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415301    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415304    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415307    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415310    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415312    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415316    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415319    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415322    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 18:16:44.460130 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415325    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415328    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415331    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415334    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415337    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415340    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415342    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415345    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415348    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415351    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415353    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415356    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415358    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415361    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415366    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415370    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415373    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415375    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415378    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 18:16:44.460721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415381    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415384    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415386    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415389    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415393    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415396    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415399    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415401    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415406    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415409    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415411    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415415    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415418    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415421    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415426    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415430    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415432    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415437    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415441    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 18:16:44.461358 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415444    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415447    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415450    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415453    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415456    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415459    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415462    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415464    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415467    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415470    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415472    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415475    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415478    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415480    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415484    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415488    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415492    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415496    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415500    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415506    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 18:16:44.461882 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415511    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415515    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415518    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415521    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415523    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415526    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415530    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415534    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415537    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415540    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415542    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415545    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415548    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.415550    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417714    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417722    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417726    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417729    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417733    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 18:16:44.462826 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417735    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417738    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417741    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417743    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417746    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417749    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417751    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417754    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417757    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417759    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417762    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417764    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417768    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417771    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417773    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417776    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417779    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417782    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417785    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417788    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 18:16:44.463379 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417790    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417793    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417796    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417798    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417801    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417803    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417806    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417808    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417811    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417815    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417819    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417821    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417824    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417827    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417829    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417832    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417835    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417837    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417840    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417842    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 18:16:44.463996 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417845    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417848    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417851    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417853    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417856    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417859    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417862    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417865    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417867    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417870    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417874    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417878    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417881    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417884    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417887    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417890    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417894    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417896    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417899    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 18:16:44.464550 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417901    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417904    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417906    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417909    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417911    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417914    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417916    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417919    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417921    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417924    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417927    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417929    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417932    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417935    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417937    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417939    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417943    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417946    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417948    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417952    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 18:16:44.465538 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417955    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417957    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417960    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417962    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417965    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417968    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417970    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417974    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417978    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417981    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417985    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417988    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417990    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417993    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.417995    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418399    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418404    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418407    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418410    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 18:16:44.533055 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418414    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 18:16:44.487230 ip-10-0-140-91 systemd[1]: Started Kubernetes Kubelet.
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418416    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418419    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418422    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418424    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418427    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418429    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418432    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418435    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418437    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418440    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418442    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418445    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418448    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418451    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418454    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418456    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418459    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418462    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 18:16:44.533994 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418465    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418468    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418470    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418473    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418475    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418478    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418481    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418483    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418486    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418490    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418494    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418497    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418500    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418503    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418506    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418509    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418513    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418516    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418520    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 18:16:44.534975 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418524    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418527    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418529    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418532    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418535    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418537    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418540    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418543    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418546    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418549    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418551    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418554    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418556    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418559    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418562    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418565    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418568    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418570    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418573    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418575    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 18:16:44.536628 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418578    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418580    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418583    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418585    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418587    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418590    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418592    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418595    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418597    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418600    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418602    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418608    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418610    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418613    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418616    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418619    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418621    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418624    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418626    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418628    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 18:16:44.537456 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418631    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418634    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418636    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418639    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418641    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418644    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418646    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418649    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418652    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418654    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418657    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418659    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418662    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418664    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418667    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418669    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.418671    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418738    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418746    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418753    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418757    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 18:16:44.538314 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418762    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418766    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418776    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418783    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418786    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418789    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418792    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418795    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418798    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418801    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418804    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418807    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418810    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418813    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418817    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418820    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418823    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418826    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418829    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418833    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418836    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418838    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418841    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418844    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 18:16:44.539340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418847    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418850    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418853    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418856    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418860    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418863    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418867    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418870    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418873    2561 flags.go:64] FLAG: --enable-server="true"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418876    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418882    2561 flags.go:64] FLAG: --event-burst="100"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418885    2561 flags.go:64] FLAG: --event-qps="50"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418888    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418891    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418894    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418898    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418901    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418904    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418907    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418909    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418912    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418916    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418919    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418922    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418925    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 18:16:44.540193 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418927    2561 flags.go:64] FLAG: --feature-gates=""
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418931    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418934    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418937    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418940    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418943    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418946    2561 flags.go:64] FLAG: --help="false"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418948    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-140-91.ec2.internal"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418952    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418954    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418957    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418960    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418964    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418968    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418971    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418974    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418977    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418982    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418985    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418988    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418991    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418993    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418997    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.418999    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 18:16:44.541196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419002    2561 flags.go:64] FLAG: --lock-file=""
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419005    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419008    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419011    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419017    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419020    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419023    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419026    2561 flags.go:64] FLAG: --logging-format="text"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419029    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419032    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419035    2561 flags.go:64] FLAG: --manifest-url=""
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419037    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419042    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419045    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419049    2561 flags.go:64] FLAG: --max-pods="110"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419052    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419055    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419057    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419060    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419063    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419066    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419070    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419078    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419081    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419083    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 18:16:44.542171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419088    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419091    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419097    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419100    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419103    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419106    2561 flags.go:64] FLAG: --port="10250"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419109    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419112    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07cd2c270e4033fe1"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419115    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419118    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419121    2561 flags.go:64] FLAG: --register-node="true"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419124    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419127    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419131    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419133    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419136    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419140    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419143    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419146    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419149    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419151    2561 flags.go:64] FLAG: --runonce="false"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419154    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419157    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419159    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419162    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 18:16:44.543102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419165    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419168    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419171    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419175    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419178    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419181    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419184    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419187    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419191    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419194    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419197    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419202    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419205    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419207    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419211    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419214    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419217    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419220    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419222    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419225    2561 flags.go:64] FLAG: --v="2"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419229    2561 flags.go:64] FLAG: --version="false"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419233    2561 flags.go:64] FLAG: --vmodule=""
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419237    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.419240    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419379    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 18:16:44.543885 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419384    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419387    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419389    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419392    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419394    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419397    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419400    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419402    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419405    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419407    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419411    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419414    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419416    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419419    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419422    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419425    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419428    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419430    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419433    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419436    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 18:16:44.544675 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419439    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419442    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419444    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419447    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419449    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419452    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419454    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419457    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419459    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419462    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419465    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419468    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419471    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419473    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419477    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419482    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419485    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419488    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419491    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 18:16:44.545536 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419494    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419497    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419499    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419503    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419506    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419509    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419512    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419514    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419519    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419521    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419524    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419527    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419530    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419532    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419535    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419537    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419540    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419543    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419545    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419548    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 18:16:44.546169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419551    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419553    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419556    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419559    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419561    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419564    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419567    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419569    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419572    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419574    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419577    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419580    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419582    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419584    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419587    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419591    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419593    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419596    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419598    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419601    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 18:16:44.546802 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419604    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419607    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419610    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419612    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419615    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419617    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419620    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419623    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419625    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419628    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419630    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419633    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419635    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419638    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419641    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419644    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419648    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419651    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 18:16:44.547450 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.419654    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.420296    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.426967    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.426982    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.427704    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.427714    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427771    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427777    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427781    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427784    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427787    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427790    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427793    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427796    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427800    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 18:16:44.548052 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427802    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427806    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427809    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427814    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427817    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427820    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427822    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427826    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427829    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427832    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427834    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427837    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427840    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427843    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427846    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427849    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427852    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427855    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427857    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427860    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 18:16:44.548681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427863    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427865    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427869    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427873    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427876    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427879    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427882    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427885    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427888    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427890    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427893    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427896    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427898    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427901    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427903    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427906    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427908    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427911    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427913    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 18:16:44.549334 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427916    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427919    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427922    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427925    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427928    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427931    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427933    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427936    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427938    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427942    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427945    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427948    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427950    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427952    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427955    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427958    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427960    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427962    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427965    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 18:16:44.550273 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427968    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427970    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427973    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427975    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427978    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427980    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427983    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427985    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427988    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427990    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427993    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427995    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.427998    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428004    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428007    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428009    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428012    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428015    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428017    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 18:16:44.550972 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428020    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428023    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428025    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428028    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428030    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428033    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428036    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428038    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428041    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428043    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428046    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428048    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428051    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.428056    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428188    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 18:16:44.552041 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428194    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428197    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428200    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428203    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428206    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428209    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428212    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428214    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428217    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428219    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428222    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428225    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428228    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428231    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428234    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428238    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428240    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428243    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428245    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428248    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 18:16:44.552602 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428251    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428254    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428257    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428260    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428263    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428266    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428268    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428271    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428273    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428276    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428279    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428295    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428298    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428302    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428305    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428308    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428311    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428314    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428317    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428320    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 18:16:44.662511 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428323    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428325    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428328    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428331    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428336    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428339    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428342    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428345    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428348    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428351    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428354    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428357    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428360    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428362    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428365    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428368    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428371    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428373    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428376    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 18:16:44.681067 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428378    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428381    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428385    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428388    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428391    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428394    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428397    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428400    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428403    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428406    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428408    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428411    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428414    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428417    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428421    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428424    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428426    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428430    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428433    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 18:16:44.682793 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428435    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428439    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428442    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428444    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428447    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428450    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428452    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428455    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428458    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428460    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428463    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428466    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428468    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428471    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428474    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428476    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428479    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428481    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428484    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 18:16:44.683873 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:44.428486    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.428491    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.428596    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.430622    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.431833    2561 server.go:1019] "Starting client certificate rotation"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.434749    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.435693    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.435739    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.438335    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.447500    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.451400    2561 log.go:25] "Validated CRI v1 image API"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.452401    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 18:16:44.684815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.456399    2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8bef8237-0755-4894-91de-06b31a81f2c7:/dev/nvme0n1p3 df662bfa-9216-43cb-bb04-4f232cc999a2:/dev/nvme0n1p4]
Jul 23 18:16:44.685768 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.456419    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 18:16:44.685995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.464700    2561 manager.go:216] Machine: {Timestamp:2026-07-23 18:16:44.463666418 +0000 UTC m=+0.311356937 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098061 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f71db294f129bbd491efb0cce9468 SystemUUID:ec2f71db-294f-129b-bd49-1efb0cce9468 BootID:f52f2bb3-07e9-4660-8183-85a680c2ad8d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8e:36:81:82:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8e:36:81:82:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:96:f4:b0:dc:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 18:16:44.685995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.464869    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 18:16:44.685995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.464990    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 18:16:44.685995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.468074    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 18:16:44.685995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.468106    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 18:16:44.685995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.468853    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 18:16:44.685995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.468870    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 18:16:44.685995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.468877    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.470096    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.471816    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.474735    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.475253    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.475374    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.475401    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.475410    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.475424    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.475434    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.475442    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.475912    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.475928    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.478351    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.480604    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.480646    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482647    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482664    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482673    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482681    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482692    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482700    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482710    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482718    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482727    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482735    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482747    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482755    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482782    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.482789    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.483931    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.483988    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.486460    2561 server.go:1268] "Started kubelet"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.486940    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.488715    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.489653    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.489691    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.492823    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.493196    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 18:16:44.687179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.497335    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ctxtl"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.503200    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.503218    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.503463    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.503903    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.503905    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.503942    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.504048    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.504058    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.504950    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ctxtl"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.510871    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.511000    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.510810    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-91.ec2.internal.18c4fdfa74a8e930  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-91.ec2.internal,UID:ip-10-0-140-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-91.ec2.internal,},FirstTimestamp:2026-07-23 18:16:44.486437168 +0000 UTC m=+0.334127688,LastTimestamp:2026-07-23 18:16:44.486437168 +0000 UTC m=+0.334127688,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-91.ec2.internal,}"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.511728    2561 factory.go:55] Registering systemd factory
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.511742    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.512495    2561 factory.go:153] Registering CRI-O factory
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.512506    2561 factory.go:223] Registration of the crio container factory successfully
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.512548    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.512566    2561 factory.go:103] Registering Raw factory
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.512577    2561 manager.go:1196] Started watching for new ooms in manager
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.513362    2561 manager.go:349] Starting recovery of all containers
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.523266    2561 manager.go:354] Recovery completed
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.530485    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.538830    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.538860    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.538874    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.539409    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.539421    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.539420    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.539439    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.540180    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.540207    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.540221    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.540673    2561 policy_none.go:49] "None policy: Start"
Jul 23 18:16:44.689069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.540690    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.540704    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.541659    2561 policy_none.go:47] "Start"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.587923    2561 manager.go:337] "Starting Device Plugin manager"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.587960    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.587974    2561 server.go:87] "Starting device plugin registration server"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.588306    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.588320    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.588473    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.588561    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.588572    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.589495    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.589547    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:44.690747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.688884    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 18:16:44.691467 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.691066    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 18:16:44.691467 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.691092    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 18:16:44.691467 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.691104    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 18:16:44.691467 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.691134    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-91.ec2.internal"
Jul 23 18:16:44.782194 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.712781    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-91.ec2.internal"
Jul 23 18:16:44.782194 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.712816    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-91.ec2.internal\": node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:44.782194 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.752999    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 18:16:44.782194 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.754316    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 18:16:44.782194 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.754375    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 18:16:44.782194 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.754402    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 18:16:44.782194 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.754449    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 18:16:44.782194 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.754561    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:44.782194 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.767122    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.857721    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"]
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.857817    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.857737    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.858867    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.858907    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.858926    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.860301    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.860483    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.860515    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.861107    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.861127    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.861134    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.861148    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.861149    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.861232    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.862734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.862757    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.863442    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.863468    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.863478    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.891931    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-91.ec2.internal\" not found" node="ip-10-0-140-91.ec2.internal"
Jul 23 18:16:44.902290 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.895935    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-91.ec2.internal\" not found" node="ip-10-0-140-91.ec2.internal"
Jul 23 18:16:44.918654 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.912528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/292e00e11710d5a1a45e5b6a5591eaca-config\") pod \"kube-apiserver-proxy-ip-10-0-140-91.ec2.internal\" (UID: \"292e00e11710d5a1a45e5b6a5591eaca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:44.918654 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.912555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9b1cd4384d54486a772889c18c0e6c72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal\" (UID: \"9b1cd4384d54486a772889c18c0e6c72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:44.918654 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:44.912581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b1cd4384d54486a772889c18c0e6c72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal\" (UID: \"9b1cd4384d54486a772889c18c0e6c72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.021925 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:44.959093    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:45.021925 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.013704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9b1cd4384d54486a772889c18c0e6c72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal\" (UID: \"9b1cd4384d54486a772889c18c0e6c72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.021925 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.013734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b1cd4384d54486a772889c18c0e6c72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal\" (UID: \"9b1cd4384d54486a772889c18c0e6c72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.021925 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.013757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/292e00e11710d5a1a45e5b6a5591eaca-config\") pod \"kube-apiserver-proxy-ip-10-0-140-91.ec2.internal\" (UID: \"292e00e11710d5a1a45e5b6a5591eaca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.021925 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.013802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/292e00e11710d5a1a45e5b6a5591eaca-config\") pod \"kube-apiserver-proxy-ip-10-0-140-91.ec2.internal\" (UID: \"292e00e11710d5a1a45e5b6a5591eaca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.021925 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.013824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b1cd4384d54486a772889c18c0e6c72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal\" (UID: \"9b1cd4384d54486a772889c18c0e6c72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.021925 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.013824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9b1cd4384d54486a772889c18c0e6c72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal\" (UID: \"9b1cd4384d54486a772889c18c0e6c72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.123192 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:45.059463    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:45.230128 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:45.162862    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:45.230128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.197541    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.230128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.199824    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.341328 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:45.263020    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:45.366952 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:45.366906    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:45.437093 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.437048    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 18:16:45.437326 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.437240    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 18:16:45.467864 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:45.467829    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-91.ec2.internal\" not found"
Jul 23 18:16:45.511034 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.510999    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 18:11:44 +0000 UTC" deadline="2028-01-29 12:25:55.832906154 +0000 UTC"
Jul 23 18:16:45.511034 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.511025    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13314h9m10.321884376s"
Jul 23 18:16:45.548730 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.548707    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 18:16:45.597665 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.597583    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 18:16:45.603871 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.603853    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.612052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.612035    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 18:16:45.613621 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.613608    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal"
Jul 23 18:16:45.626765 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.626743    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 18:16:45.638187 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.638168    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 18:16:45.779301 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:45.779265    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292e00e11710d5a1a45e5b6a5591eaca.slice/crio-bad6a2235c64e4ad9196890e5b36103c5c8ec7349a759e02a22324a9dd848c4a WatchSource:0}: Error finding container bad6a2235c64e4ad9196890e5b36103c5c8ec7349a759e02a22324a9dd848c4a: Status 404 returned error can't find the container with id bad6a2235c64e4ad9196890e5b36103c5c8ec7349a759e02a22324a9dd848c4a
Jul 23 18:16:45.783470 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:45.783452    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:16:46.012301 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.012263    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1cd4384d54486a772889c18c0e6c72.slice/crio-0f8bafac3ae798e3137162c4edbea1a5732ebdbc02dacd4ab71dd35e1ff3b114 WatchSource:0}: Error finding container 0f8bafac3ae798e3137162c4edbea1a5732ebdbc02dacd4ab71dd35e1ff3b114: Status 404 returned error can't find the container with id 0f8bafac3ae798e3137162c4edbea1a5732ebdbc02dacd4ab71dd35e1ff3b114
Jul 23 18:16:46.477642 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.477562    2561 apiserver.go:52] "Watching apiserver"
Jul 23 18:16:46.486457 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.486428    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 18:16:46.487872 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.487837    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-fm4gn","kube-system/global-pull-secret-syncer-vqch6","kube-system/konnectivity-agent-j6gpl","kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal","openshift-image-registry/node-ca-rk2rs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal","openshift-multus/multus-h86v2","openshift-network-operator/iptables-alerter-wr5zm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn","openshift-cluster-node-tuning-operator/tuned-5brsd","openshift-dns/node-resolver-v4g5k","openshift-multus/multus-additional-cni-plugins-d44gh","openshift-multus/network-metrics-daemon-75jnc","openshift-network-diagnostics/network-check-target-kwpb4"]
Jul 23 18:16:46.489782 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.489751    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.491259 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.491237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:46.491402 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.491328    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:16:46.492701 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.492679    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sr27z\""
Jul 23 18:16:46.492701 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.492691    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 18:16:46.492858 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.492783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:16:46.493641 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.493469    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 18:16:46.493641 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.493479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 18:16:46.493641 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.493492    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 18:16:46.493641 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.493484    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 18:16:46.493641 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.493584    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 18:16:46.494281 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.494258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.495171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.495149    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 18:16:46.495334 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.495318    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k72gt\""
Jul 23 18:16:46.495463 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.495447    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 18:16:46.496121 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.496103    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.499395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.497131    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 18:16:46.499395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.497206    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 18:16:46.499395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.497135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 18:16:46.499395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.497918    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lv6zs\""
Jul 23 18:16:46.499395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.498827    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 18:16:46.499395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.498990    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 18:16:46.499395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.499207    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 18:16:46.499395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.499237    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 18:16:46.499395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.499238    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s7skp\""
Jul 23 18:16:46.500724 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.500384    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.502304 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.502276    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.502694 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.502659    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hspk7\""
Jul 23 18:16:46.503243 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.503223    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 18:16:46.503352 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.503291    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 18:16:46.503352 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.503309    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 18:16:46.503717 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.503698    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 18:16:46.503823 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.503785    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.504788 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.504766    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 18:16:46.504885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.504800    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 18:16:46.505081 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.505060    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4g4k\""
Jul 23 18:16:46.505179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.505100    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 18:16:46.505404 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.505387    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.506118 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.506091    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6vgps\""
Jul 23 18:16:46.506498 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.506473    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 18:16:46.506598 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.506571    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 18:16:46.507215 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.507190    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.508347 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.507718    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4lsqb\""
Jul 23 18:16:46.508347 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.507757    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 18:16:46.508347 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.508260    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 18:16:46.509107 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.509016    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:46.509107 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.509092    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:16:46.509787 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.509561    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 18:16:46.509787 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.509708    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 18:16:46.509787 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.509757    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xp5hv\""
Jul 23 18:16:46.510305 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.510276    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:46.510386 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.510356    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:16:46.519232 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.519194    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 18:16:46.523797 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.523775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40b7b2b4-947f-42bf-9f83-6bd57c73251e-serviceca\") pod \"node-ca-rk2rs\" (UID: \"40b7b2b4-947f-42bf-9f83-6bd57c73251e\") " pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.523918 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.523806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-var-lib-cni-multus\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.523918 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.523827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-conf-dir\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.523918 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.523847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-run\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.523918 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.523871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-run-k8s-cni-cncf-io\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.524247 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.523937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-var-lib-cni-bin\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.524395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfs6t\" (UniqueName: \"kubernetes.io/projected/307a0a42-dfca-4ad1-a5e2-0afa874975a4-kube-api-access-dfs6t\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.524477 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-cni-netd\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.524477 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-var-lib-kubelet\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.524589 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-node-log\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.524589 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9052287-a150-41b0-b0ef-82313217f2e7-ovn-node-metrics-cert\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.524589 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524526    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9052287-a150-41b0-b0ef-82313217f2e7-ovnkube-script-lib\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.524589 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfp4s\" (UniqueName: \"kubernetes.io/projected/d9052287-a150-41b0-b0ef-82313217f2e7-kube-api-access-sfp4s\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.524589 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e804a5b4-b905-4966-a132-976a3d67e6dc-tmp-dir\") pod \"node-resolver-v4g5k\" (UID: \"e804a5b4-b905-4966-a132-976a3d67e6dc\") " pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.524813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8w9\" (UniqueName: \"kubernetes.io/projected/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-kube-api-access-6p8w9\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.524813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524637    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcad1db1-cdf1-465a-bb2b-05dfda66096e-host-slash\") pod \"iptables-alerter-wr5zm\" (UID: \"dcad1db1-cdf1-465a-bb2b-05dfda66096e\") " pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.524813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524670    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39bf1855-8430-4c11-a577-32be87bf9fa5-agent-certs\") pod \"konnectivity-agent-j6gpl\" (UID: \"39bf1855-8430-4c11-a577-32be87bf9fa5\") " pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:16:46.524813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-cni-dir\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.524813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-etc-kubernetes\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.524813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-etc-openvswitch\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.524813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcad1db1-cdf1-465a-bb2b-05dfda66096e-iptables-alerter-script\") pod \"iptables-alerter-wr5zm\" (UID: \"dcad1db1-cdf1-465a-bb2b-05dfda66096e\") " pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.524813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-sysctl-d\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzjlt\" (UniqueName: \"kubernetes.io/projected/40b7b2b4-947f-42bf-9f83-6bd57c73251e-kube-api-access-gzjlt\") pod \"node-ca-rk2rs\" (UID: \"40b7b2b4-947f-42bf-9f83-6bd57c73251e\") " pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-os-release\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-var-lib-kubelet\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-kubelet\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-var-lib-openvswitch\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.524979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-sysconfig\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-lib-modules\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ee12eab-d883-42ae-94cc-6618823fad3c-kubelet-config\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-daemon-config\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.525128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525151    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-systemd\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvfw\" (UniqueName: \"kubernetes.io/projected/23960491-7757-4a97-8f6f-445e6c5daac5-kube-api-access-cdvfw\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39bf1855-8430-4c11-a577-32be87bf9fa5-konnectivity-ca\") pod \"konnectivity-agent-j6gpl\" (UID: \"39bf1855-8430-4c11-a577-32be87bf9fa5\") " pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-cni-binary-copy\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-run-openvswitch\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-log-socket\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9052287-a150-41b0-b0ef-82313217f2e7-env-overrides\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23960491-7757-4a97-8f6f-445e6c5daac5-tmp\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e804a5b4-b905-4966-a132-976a3d67e6dc-hosts-file\") pod \"node-resolver-v4g5k\" (UID: \"e804a5b4-b905-4966-a132-976a3d67e6dc\") " pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvcfs\" (UniqueName: \"kubernetes.io/projected/e804a5b4-b905-4966-a132-976a3d67e6dc-kube-api-access-hvcfs\") pod \"node-resolver-v4g5k\" (UID: \"e804a5b4-b905-4966-a132-976a3d67e6dc\") " pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-run-multus-certs\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-socket-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-systemd-units\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.525710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-cni-bin\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525597    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-sysctl-conf\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-hostroot\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-device-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-run-netns\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-run-systemd\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tptj\" (UniqueName: \"kubernetes.io/projected/dcad1db1-cdf1-465a-bb2b-05dfda66096e-kube-api-access-5tptj\") pod \"iptables-alerter-wr5zm\" (UID: \"dcad1db1-cdf1-465a-bb2b-05dfda66096e\") " pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-modprobe-d\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40b7b2b4-947f-42bf-9f83-6bd57c73251e-host\") pod \"node-ca-rk2rs\" (UID: \"40b7b2b4-947f-42bf-9f83-6bd57c73251e\") " pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-socket-dir-parent\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-registration-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-sys-fs\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-run-ovn\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.526226 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9052287-a150-41b0-b0ef-82313217f2e7-ovnkube-config\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-kubernetes\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-host\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23960491-7757-4a97-8f6f-445e6c5daac5-etc-tuned\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-cnibin\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.525966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-run-netns\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.526005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-slash\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.526021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-system-cni-dir\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.526037    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-etc-selinux\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.526065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-sys\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.526889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.526097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ee12eab-d883-42ae-94cc-6618823fad3c-dbus\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:46.555547 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.555519    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxhkj"
Jul 23 18:16:46.567257 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.567179    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxhkj"
Jul 23 18:16:46.604933 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.604904    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 18:16:46.626889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.626857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-run-openvswitch\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.627052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.626900    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-log-socket\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.627052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.626927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9052287-a150-41b0-b0ef-82313217f2e7-env-overrides\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.627052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.626972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23960491-7757-4a97-8f6f-445e6c5daac5-tmp\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.627052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.626981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-run-openvswitch\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.627052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e804a5b4-b905-4966-a132-976a3d67e6dc-hosts-file\") pod \"node-resolver-v4g5k\" (UID: \"e804a5b4-b905-4966-a132-976a3d67e6dc\") " pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.627052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcfs\" (UniqueName: \"kubernetes.io/projected/e804a5b4-b905-4966-a132-976a3d67e6dc-kube-api-access-hvcfs\") pod \"node-resolver-v4g5k\" (UID: \"e804a5b4-b905-4966-a132-976a3d67e6dc\") " pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-run-multus-certs\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-socket-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-systemd-units\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-cni-bin\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-sysctl-conf\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-hostroot\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627254    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-device-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-run-netns\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627271    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-socket-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.627394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627383    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e804a5b4-b905-4966-a132-976a3d67e6dc-hosts-file\") pod \"node-resolver-v4g5k\" (UID: \"e804a5b4-b905-4966-a132-976a3d67e6dc\") " pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-run-systemd\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-cni-bin\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627421    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9052287-a150-41b0-b0ef-82313217f2e7-env-overrides\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627427    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-device-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-hostroot\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-run-multus-certs\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627364    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-run-systemd\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-log-socket\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tptj\" (UniqueName: \"kubernetes.io/projected/dcad1db1-cdf1-465a-bb2b-05dfda66096e-kube-api-access-5tptj\") pod \"iptables-alerter-wr5zm\" (UID: \"dcad1db1-cdf1-465a-bb2b-05dfda66096e\") " pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-modprobe-d\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40b7b2b4-947f-42bf-9f83-6bd57c73251e-host\") pod \"node-ca-rk2rs\" (UID: \"40b7b2b4-947f-42bf-9f83-6bd57c73251e\") " pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-socket-dir-parent\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-registration-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-sys-fs\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.628032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627644    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-systemd-units\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-run-ovn\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627721    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-sysctl-conf\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-run-ovn\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627762    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-sys-fs\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627755    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-registration-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40b7b2b4-947f-42bf-9f83-6bd57c73251e-host\") pod \"node-ca-rk2rs\" (UID: \"40b7b2b4-947f-42bf-9f83-6bd57c73251e\") " pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-modprobe-d\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9052287-a150-41b0-b0ef-82313217f2e7-ovnkube-config\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-kubernetes\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-socket-dir-parent\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-kubernetes\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-host\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-host\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23960491-7757-4a97-8f6f-445e6c5daac5-etc-tuned\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.628803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.627973    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-cnibin\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628005    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-run-netns\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-slash\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-cnibin\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-system-cni-dir\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-etc-selinux\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-slash\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a8557ac-27ae-4f77-8c7a-1d32266f570e-cni-binary-copy\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-system-cni-dir\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-run-netns\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a8557ac-27ae-4f77-8c7a-1d32266f570e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-sys\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-etc-selinux\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628242    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ee12eab-d883-42ae-94cc-6618823fad3c-dbus\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:46.629584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40b7b2b4-947f-42bf-9f83-6bd57c73251e-serviceca\") pod \"node-ca-rk2rs\" (UID: \"40b7b2b4-947f-42bf-9f83-6bd57c73251e\") " pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9052287-a150-41b0-b0ef-82313217f2e7-ovnkube-config\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-var-lib-cni-multus\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-sys\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-conf-dir\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpb2p\" (UniqueName: \"kubernetes.io/projected/8a8557ac-27ae-4f77-8c7a-1d32266f570e-kube-api-access-tpb2p\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-var-lib-cni-multus\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-run\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ee12eab-d883-42ae-94cc-6618823fad3c-dbus\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628438    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-conf-dir\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628462    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-run-k8s-cni-cncf-io\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-run\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628485    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-var-lib-cni-bin\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfs6t\" (UniqueName: \"kubernetes.io/projected/307a0a42-dfca-4ad1-a5e2-0afa874975a4-kube-api-access-dfs6t\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-cni-netd\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.630327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628543    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-var-lib-cni-bin\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-var-lib-kubelet\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-os-release\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-cni-netd\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-run-k8s-cni-cncf-io\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-node-log\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9052287-a150-41b0-b0ef-82313217f2e7-ovn-node-metrics-cert\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9052287-a150-41b0-b0ef-82313217f2e7-ovnkube-script-lib\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-var-lib-kubelet\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-node-log\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfp4s\" (UniqueName: \"kubernetes.io/projected/d9052287-a150-41b0-b0ef-82313217f2e7-kube-api-access-sfp4s\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e804a5b4-b905-4966-a132-976a3d67e6dc-tmp-dir\") pod \"node-resolver-v4g5k\" (UID: \"e804a5b4-b905-4966-a132-976a3d67e6dc\") " pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628736    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/40b7b2b4-947f-42bf-9f83-6bd57c73251e-serviceca\") pod \"node-ca-rk2rs\" (UID: \"40b7b2b4-947f-42bf-9f83-6bd57c73251e\") " pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8w9\" (UniqueName: \"kubernetes.io/projected/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-kube-api-access-6p8w9\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcad1db1-cdf1-465a-bb2b-05dfda66096e-host-slash\") pod \"iptables-alerter-wr5zm\" (UID: \"dcad1db1-cdf1-465a-bb2b-05dfda66096e\") " pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.631151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39bf1855-8430-4c11-a577-32be87bf9fa5-agent-certs\") pod \"konnectivity-agent-j6gpl\" (UID: \"39bf1855-8430-4c11-a577-32be87bf9fa5\") " pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-cni-dir\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-etc-kubernetes\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-etc-openvswitch\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcad1db1-cdf1-465a-bb2b-05dfda66096e-iptables-alerter-script\") pod \"iptables-alerter-wr5zm\" (UID: \"dcad1db1-cdf1-465a-bb2b-05dfda66096e\") " pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-sysctl-d\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzjlt\" (UniqueName: \"kubernetes.io/projected/40b7b2b4-947f-42bf-9f83-6bd57c73251e-kube-api-access-gzjlt\") pod \"node-ca-rk2rs\" (UID: \"40b7b2b4-947f-42bf-9f83-6bd57c73251e\") " pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-os-release\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-etc-kubernetes\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-var-lib-kubelet\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e804a5b4-b905-4966-a132-976a3d67e6dc-tmp-dir\") pod \"node-resolver-v4g5k\" (UID: \"e804a5b4-b905-4966-a132-976a3d67e6dc\") " pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcad1db1-cdf1-465a-bb2b-05dfda66096e-host-slash\") pod \"iptables-alerter-wr5zm\" (UID: \"dcad1db1-cdf1-465a-bb2b-05dfda66096e\") " pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629130    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-etc-openvswitch\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629147    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-kubelet\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.631906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-kubelet\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-var-lib-openvswitch\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-sysconfig\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9052287-a150-41b0-b0ef-82313217f2e7-ovnkube-script-lib\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-lib-modules\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-sysctl-d\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-sysconfig\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-lib-modules\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-cni-dir\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.629331    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629336    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-var-lib-openvswitch\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-system-cni-dir\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a8557ac-27ae-4f77-8c7a-1d32266f570e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-os-release\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.632689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-host-var-lib-kubelet\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.629446    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret podName:4ee12eab-d883-42ae-94cc-6618823fad3c nodeName:}" failed. No retries permitted until 2026-07-23 18:16:47.129406851 +0000 UTC m=+2.977097381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret") pod "global-pull-secret-syncer-vqch6" (UID: "4ee12eab-d883-42ae-94cc-6618823fad3c") : object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcggg\" (UniqueName: \"kubernetes.io/projected/4e9413cd-4d10-4609-a427-55fcc8474d38-kube-api-access-mcggg\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.628356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-run-netns\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ee12eab-d883-42ae-94cc-6618823fad3c-kubelet-config\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-daemon-config\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-systemd\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ee12eab-d883-42ae-94cc-6618823fad3c-kubelet-config\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvfw\" (UniqueName: \"kubernetes.io/projected/23960491-7757-4a97-8f6f-445e6c5daac5-kube-api-access-cdvfw\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39bf1855-8430-4c11-a577-32be87bf9fa5-konnectivity-ca\") pod \"konnectivity-agent-j6gpl\" (UID: \"39bf1855-8430-4c11-a577-32be87bf9fa5\") " pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-cni-binary-copy\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.633328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23960491-7757-4a97-8f6f-445e6c5daac5-etc-systemd\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-cnibin\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629809    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.629708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/307a0a42-dfca-4ad1-a5e2-0afa874975a4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.630182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcad1db1-cdf1-465a-bb2b-05dfda66096e-iptables-alerter-script\") pod \"iptables-alerter-wr5zm\" (UID: \"dcad1db1-cdf1-465a-bb2b-05dfda66096e\") " pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.630193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-multus-daemon-config\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.630274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39bf1855-8430-4c11-a577-32be87bf9fa5-konnectivity-ca\") pod \"konnectivity-agent-j6gpl\" (UID: \"39bf1855-8430-4c11-a577-32be87bf9fa5\") " pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.630276    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9052287-a150-41b0-b0ef-82313217f2e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.631107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-cni-binary-copy\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.632615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23960491-7757-4a97-8f6f-445e6c5daac5-tmp\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.632751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9052287-a150-41b0-b0ef-82313217f2e7-ovn-node-metrics-cert\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.632785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39bf1855-8430-4c11-a577-32be87bf9fa5-agent-certs\") pod \"konnectivity-agent-j6gpl\" (UID: \"39bf1855-8430-4c11-a577-32be87bf9fa5\") " pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:16:46.633924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.632790    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23960491-7757-4a97-8f6f-445e6c5daac5-etc-tuned\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.637058 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.636992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvcfs\" (UniqueName: \"kubernetes.io/projected/e804a5b4-b905-4966-a132-976a3d67e6dc-kube-api-access-hvcfs\") pod \"node-resolver-v4g5k\" (UID: \"e804a5b4-b905-4966-a132-976a3d67e6dc\") " pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.637221 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.637196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tptj\" (UniqueName: \"kubernetes.io/projected/dcad1db1-cdf1-465a-bb2b-05dfda66096e-kube-api-access-5tptj\") pod \"iptables-alerter-wr5zm\" (UID: \"dcad1db1-cdf1-465a-bb2b-05dfda66096e\") " pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.637731 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.637705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfp4s\" (UniqueName: \"kubernetes.io/projected/d9052287-a150-41b0-b0ef-82313217f2e7-kube-api-access-sfp4s\") pod \"ovnkube-node-fm4gn\" (UID: \"d9052287-a150-41b0-b0ef-82313217f2e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.637893 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.637866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8w9\" (UniqueName: \"kubernetes.io/projected/abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb-kube-api-access-6p8w9\") pod \"multus-h86v2\" (UID: \"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb\") " pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.638744 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.638725    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfs6t\" (UniqueName: \"kubernetes.io/projected/307a0a42-dfca-4ad1-a5e2-0afa874975a4-kube-api-access-dfs6t\") pod \"aws-ebs-csi-driver-node-wscnn\" (UID: \"307a0a42-dfca-4ad1-a5e2-0afa874975a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.640513 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.640487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzjlt\" (UniqueName: \"kubernetes.io/projected/40b7b2b4-947f-42bf-9f83-6bd57c73251e-kube-api-access-gzjlt\") pod \"node-ca-rk2rs\" (UID: \"40b7b2b4-947f-42bf-9f83-6bd57c73251e\") " pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.641919 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.641891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvfw\" (UniqueName: \"kubernetes.io/projected/23960491-7757-4a97-8f6f-445e6c5daac5-kube-api-access-cdvfw\") pod \"tuned-5brsd\" (UID: \"23960491-7757-4a97-8f6f-445e6c5daac5\") " pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.730691 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-cnibin\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.730691 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:46.730691 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.730691 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a8557ac-27ae-4f77-8c7a-1d32266f570e-cni-binary-copy\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731014 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-cnibin\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731014 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a8557ac-27ae-4f77-8c7a-1d32266f570e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731014 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:46.731014 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpb2p\" (UniqueName: \"kubernetes.io/projected/8a8557ac-27ae-4f77-8c7a-1d32266f570e-kube-api-access-tpb2p\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731014 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-os-release\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731014 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730896    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731014 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.730982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-os-release\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731014 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.731009    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:46.731450 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.731086    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs podName:4e9413cd-4d10-4609-a427-55fcc8474d38 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:47.231065608 +0000 UTC m=+3.078756119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs") pod "network-metrics-daemon-75jnc" (UID: "4e9413cd-4d10-4609-a427-55fcc8474d38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:46.731450 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.731193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-system-cni-dir\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731450 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.731235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a8557ac-27ae-4f77-8c7a-1d32266f570e-system-cni-dir\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731450 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.731237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a8557ac-27ae-4f77-8c7a-1d32266f570e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731450 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.731263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a8557ac-27ae-4f77-8c7a-1d32266f570e-cni-binary-copy\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731450 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.731282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcggg\" (UniqueName: \"kubernetes.io/projected/4e9413cd-4d10-4609-a427-55fcc8474d38-kube-api-access-mcggg\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:46.731714 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.731458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a8557ac-27ae-4f77-8c7a-1d32266f570e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.731714 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.731657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a8557ac-27ae-4f77-8c7a-1d32266f570e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.739394 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.739367    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 18:16:46.739394 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.739393    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 18:16:46.739570 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.739407    2561 projected.go:196] Error preparing data for projected volume kube-api-access-n7kn7 for pod openshift-network-diagnostics/network-check-target-kwpb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:46.739570 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:46.739481    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7 podName:a4abfb91-e90d-450f-8a07-b6fa81d7a6d0 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:47.239462816 +0000 UTC m=+3.087153328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n7kn7" (UniqueName: "kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7") pod "network-check-target-kwpb4" (UID: "a4abfb91-e90d-450f-8a07-b6fa81d7a6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:46.742792 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.742760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpb2p\" (UniqueName: \"kubernetes.io/projected/8a8557ac-27ae-4f77-8c7a-1d32266f570e-kube-api-access-tpb2p\") pod \"multus-additional-cni-plugins-d44gh\" (UID: \"8a8557ac-27ae-4f77-8c7a-1d32266f570e\") " pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.742888 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.742825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcggg\" (UniqueName: \"kubernetes.io/projected/4e9413cd-4d10-4609-a427-55fcc8474d38-kube-api-access-mcggg\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:46.759364 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.759311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal" event={"ID":"9b1cd4384d54486a772889c18c0e6c72","Type":"ContainerStarted","Data":"0f8bafac3ae798e3137162c4edbea1a5732ebdbc02dacd4ab71dd35e1ff3b114"}
Jul 23 18:16:46.760518 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.760485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal" event={"ID":"292e00e11710d5a1a45e5b6a5591eaca","Type":"ContainerStarted","Data":"bad6a2235c64e4ad9196890e5b36103c5c8ec7349a759e02a22324a9dd848c4a"}
Jul 23 18:16:46.803806 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.803774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:16:46.812197 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.812170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:16:46.816249 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.816220    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9052287_a150_41b0_b0ef_82313217f2e7.slice/crio-0de3342880b8dc71d0fb21b610b82bc716cac56d332d6df3d5c99e0305896815 WatchSource:0}: Error finding container 0de3342880b8dc71d0fb21b610b82bc716cac56d332d6df3d5c99e0305896815: Status 404 returned error can't find the container with id 0de3342880b8dc71d0fb21b610b82bc716cac56d332d6df3d5c99e0305896815
Jul 23 18:16:46.821803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.821778    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rk2rs"
Jul 23 18:16:46.824487 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.824465    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bf1855_8430_4c11_a577_32be87bf9fa5.slice/crio-5a46f597ab9ecbaa912f32c8bc58d0eba1647d8a4c7651b8a556a4109ea993a3 WatchSource:0}: Error finding container 5a46f597ab9ecbaa912f32c8bc58d0eba1647d8a4c7651b8a556a4109ea993a3: Status 404 returned error can't find the container with id 5a46f597ab9ecbaa912f32c8bc58d0eba1647d8a4c7651b8a556a4109ea993a3
Jul 23 18:16:46.828309 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.828286    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h86v2"
Jul 23 18:16:46.830661 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.830628    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b7b2b4_947f_42bf_9f83_6bd57c73251e.slice/crio-d57f7f8b1d9ddb73e8fcd20fac312adc2cf4adae2e5927e47abea426bb57e40a WatchSource:0}: Error finding container d57f7f8b1d9ddb73e8fcd20fac312adc2cf4adae2e5927e47abea426bb57e40a: Status 404 returned error can't find the container with id d57f7f8b1d9ddb73e8fcd20fac312adc2cf4adae2e5927e47abea426bb57e40a
Jul 23 18:16:46.835190 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.835166    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wr5zm"
Jul 23 18:16:46.837328 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.837283    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc48b6f_98c5_43b6_8a1e_edd9d21e0ccb.slice/crio-65e1539e1dd22ea05ca153cd59819ba2ff441c512306aa8942e7a0510aa1bb24 WatchSource:0}: Error finding container 65e1539e1dd22ea05ca153cd59819ba2ff441c512306aa8942e7a0510aa1bb24: Status 404 returned error can't find the container with id 65e1539e1dd22ea05ca153cd59819ba2ff441c512306aa8942e7a0510aa1bb24
Jul 23 18:16:46.843456 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.843435    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn"
Jul 23 18:16:46.843615 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.843580    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcad1db1_cdf1_465a_bb2b_05dfda66096e.slice/crio-e5ead2c67b6d0511d58bb47013c47fab581bde57536a2ddf2547b02987e73b99 WatchSource:0}: Error finding container e5ead2c67b6d0511d58bb47013c47fab581bde57536a2ddf2547b02987e73b99: Status 404 returned error can't find the container with id e5ead2c67b6d0511d58bb47013c47fab581bde57536a2ddf2547b02987e73b99
Jul 23 18:16:46.851681 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.851658    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5brsd"
Jul 23 18:16:46.851936 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.851908    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307a0a42_dfca_4ad1_a5e2_0afa874975a4.slice/crio-c92e525543999d01b2dde0072f180084789bc27815a910c592e400f262dedbd5 WatchSource:0}: Error finding container c92e525543999d01b2dde0072f180084789bc27815a910c592e400f262dedbd5: Status 404 returned error can't find the container with id c92e525543999d01b2dde0072f180084789bc27815a910c592e400f262dedbd5
Jul 23 18:16:46.858803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.858782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v4g5k"
Jul 23 18:16:46.860164 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.860085    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23960491_7757_4a97_8f6f_445e6c5daac5.slice/crio-4790d45134c3baf48d93adfa6b1d40ca3e7987e8af380538572b2b071a91efcf WatchSource:0}: Error finding container 4790d45134c3baf48d93adfa6b1d40ca3e7987e8af380538572b2b071a91efcf: Status 404 returned error can't find the container with id 4790d45134c3baf48d93adfa6b1d40ca3e7987e8af380538572b2b071a91efcf
Jul 23 18:16:46.864208 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:46.864181    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d44gh"
Jul 23 18:16:46.867574 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.867542    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode804a5b4_b905_4966_a132_976a3d67e6dc.slice/crio-9b2f60a198a11ebfaeeb4c533f9aaf7de6b8bf1a22bacfec431790765776d0d8 WatchSource:0}: Error finding container 9b2f60a198a11ebfaeeb4c533f9aaf7de6b8bf1a22bacfec431790765776d0d8: Status 404 returned error can't find the container with id 9b2f60a198a11ebfaeeb4c533f9aaf7de6b8bf1a22bacfec431790765776d0d8
Jul 23 18:16:46.873453 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:16:46.873427    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8557ac_27ae_4f77_8c7a_1d32266f570e.slice/crio-add7608453bc0a1807baa54f967423f78a6552f76545708288789de18da05724 WatchSource:0}: Error finding container add7608453bc0a1807baa54f967423f78a6552f76545708288789de18da05724: Status 404 returned error can't find the container with id add7608453bc0a1807baa54f967423f78a6552f76545708288789de18da05724
Jul 23 18:16:47.007013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.006929    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 18:16:47.134007 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.133968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:47.134203 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:47.134137    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:47.134274 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:47.134221    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret podName:4ee12eab-d883-42ae-94cc-6618823fad3c nodeName:}" failed. No retries permitted until 2026-07-23 18:16:48.134198137 +0000 UTC m=+3.981888645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret") pod "global-pull-secret-syncer-vqch6" (UID: "4ee12eab-d883-42ae-94cc-6618823fad3c") : object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:47.234959 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.234907    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:47.235162 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:47.235091    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:47.235253 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:47.235180    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs podName:4e9413cd-4d10-4609-a427-55fcc8474d38 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:48.235158022 +0000 UTC m=+4.082848557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs") pod "network-metrics-daemon-75jnc" (UID: "4e9413cd-4d10-4609-a427-55fcc8474d38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:47.335599 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.335560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:47.335753 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:47.335720    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 18:16:47.335753 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:47.335738    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 18:16:47.335753 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:47.335747    2561 projected.go:196] Error preparing data for projected volume kube-api-access-n7kn7 for pod openshift-network-diagnostics/network-check-target-kwpb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:47.335911 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:47.335803    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7 podName:a4abfb91-e90d-450f-8a07-b6fa81d7a6d0 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:48.33578289 +0000 UTC m=+4.183473398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7kn7" (UniqueName: "kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7") pod "network-check-target-kwpb4" (UID: "a4abfb91-e90d-450f-8a07-b6fa81d7a6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:47.570690 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.570452    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 18:11:46 +0000 UTC" deadline="2028-04-25 07:24:21.52991381 +0000 UTC"
Jul 23 18:16:47.570690 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.570496    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15397h7m33.959422982s"
Jul 23 18:16:47.755888 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.755857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:47.756020 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:47.755995    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:16:47.783296 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.783257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d44gh" event={"ID":"8a8557ac-27ae-4f77-8c7a-1d32266f570e","Type":"ContainerStarted","Data":"add7608453bc0a1807baa54f967423f78a6552f76545708288789de18da05724"}
Jul 23 18:16:47.796212 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.796176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v4g5k" event={"ID":"e804a5b4-b905-4966-a132-976a3d67e6dc","Type":"ContainerStarted","Data":"9b2f60a198a11ebfaeeb4c533f9aaf7de6b8bf1a22bacfec431790765776d0d8"}
Jul 23 18:16:47.811662 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.811624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wr5zm" event={"ID":"dcad1db1-cdf1-465a-bb2b-05dfda66096e","Type":"ContainerStarted","Data":"e5ead2c67b6d0511d58bb47013c47fab581bde57536a2ddf2547b02987e73b99"}
Jul 23 18:16:47.821882 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.821849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h86v2" event={"ID":"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb","Type":"ContainerStarted","Data":"65e1539e1dd22ea05ca153cd59819ba2ff441c512306aa8942e7a0510aa1bb24"}
Jul 23 18:16:47.848273 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.848151    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rk2rs" event={"ID":"40b7b2b4-947f-42bf-9f83-6bd57c73251e","Type":"ContainerStarted","Data":"d57f7f8b1d9ddb73e8fcd20fac312adc2cf4adae2e5927e47abea426bb57e40a"}
Jul 23 18:16:47.862896 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.862571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6gpl" event={"ID":"39bf1855-8430-4c11-a577-32be87bf9fa5","Type":"ContainerStarted","Data":"5a46f597ab9ecbaa912f32c8bc58d0eba1647d8a4c7651b8a556a4109ea993a3"}
Jul 23 18:16:47.883382 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.883284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerStarted","Data":"0de3342880b8dc71d0fb21b610b82bc716cac56d332d6df3d5c99e0305896815"}
Jul 23 18:16:47.893094 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.893059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5brsd" event={"ID":"23960491-7757-4a97-8f6f-445e6c5daac5","Type":"ContainerStarted","Data":"4790d45134c3baf48d93adfa6b1d40ca3e7987e8af380538572b2b071a91efcf"}
Jul 23 18:16:47.900050 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:47.899964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn" event={"ID":"307a0a42-dfca-4ad1-a5e2-0afa874975a4","Type":"ContainerStarted","Data":"c92e525543999d01b2dde0072f180084789bc27815a910c592e400f262dedbd5"}
Jul 23 18:16:48.142273 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:48.142184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:48.142463 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.142345    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:48.142463 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.142413    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret podName:4ee12eab-d883-42ae-94cc-6618823fad3c nodeName:}" failed. No retries permitted until 2026-07-23 18:16:50.142393394 +0000 UTC m=+5.990083925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret") pod "global-pull-secret-syncer-vqch6" (UID: "4ee12eab-d883-42ae-94cc-6618823fad3c") : object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:48.242626 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:48.242584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:48.242816 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.242799    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:48.242899 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.242869    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs podName:4e9413cd-4d10-4609-a427-55fcc8474d38 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:50.242848757 +0000 UTC m=+6.090539269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs") pod "network-metrics-daemon-75jnc" (UID: "4e9413cd-4d10-4609-a427-55fcc8474d38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:48.343736 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:48.343682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:48.343940 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.343885    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 18:16:48.343940 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.343909    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 18:16:48.343940 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.343924    2561 projected.go:196] Error preparing data for projected volume kube-api-access-n7kn7 for pod openshift-network-diagnostics/network-check-target-kwpb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:48.344102 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.343988    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7 podName:a4abfb91-e90d-450f-8a07-b6fa81d7a6d0 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:50.343969992 +0000 UTC m=+6.191660504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7kn7" (UniqueName: "kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7") pod "network-check-target-kwpb4" (UID: "a4abfb91-e90d-450f-8a07-b6fa81d7a6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:48.571015 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:48.570922    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 18:11:46 +0000 UTC" deadline="2028-04-28 17:36:48.840464746 +0000 UTC"
Jul 23 18:16:48.571015 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:48.570967    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15479h20m0.26950316s"
Jul 23 18:16:48.755909 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:48.755214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:48.755909 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:48.755445    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:48.755909 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.755546    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:16:48.756334 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:48.756284    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:16:48.931416 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:48.930440    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b1cd4384d54486a772889c18c0e6c72" containerID="2d1324b17227a08184fb1a5166f461f924170dd359ded8721c7ec6a4875715a3" exitCode=0
Jul 23 18:16:48.931416 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:48.930506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal" event={"ID":"9b1cd4384d54486a772889c18c0e6c72","Type":"ContainerDied","Data":"2d1324b17227a08184fb1a5166f461f924170dd359ded8721c7ec6a4875715a3"}
Jul 23 18:16:49.755309 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:49.755024    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:49.755825 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:49.755427    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:16:49.962720 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:49.962678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal" event={"ID":"9b1cd4384d54486a772889c18c0e6c72","Type":"ContainerStarted","Data":"88fa99c42a1c9d7a1e111b20d7d65e28b5e4b472b5cb731029d951c775d6dc08"}
Jul 23 18:16:50.160422 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:50.160341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:50.160584 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.160523    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:50.160638 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.160585    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret podName:4ee12eab-d883-42ae-94cc-6618823fad3c nodeName:}" failed. No retries permitted until 2026-07-23 18:16:54.160567074 +0000 UTC m=+10.008257579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret") pod "global-pull-secret-syncer-vqch6" (UID: "4ee12eab-d883-42ae-94cc-6618823fad3c") : object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:50.261346 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:50.261306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:50.261538 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.261465    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:50.261538 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.261529    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs podName:4e9413cd-4d10-4609-a427-55fcc8474d38 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:54.261510259 +0000 UTC m=+10.109200770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs") pod "network-metrics-daemon-75jnc" (UID: "4e9413cd-4d10-4609-a427-55fcc8474d38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:50.362498 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:50.361771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:50.362498 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.362005    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 18:16:50.362498 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.362029    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 18:16:50.362498 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.362043    2561 projected.go:196] Error preparing data for projected volume kube-api-access-n7kn7 for pod openshift-network-diagnostics/network-check-target-kwpb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:50.362498 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.362114    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7 podName:a4abfb91-e90d-450f-8a07-b6fa81d7a6d0 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:54.362095761 +0000 UTC m=+10.209786289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7kn7" (UniqueName: "kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7") pod "network-check-target-kwpb4" (UID: "a4abfb91-e90d-450f-8a07-b6fa81d7a6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:50.755310 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:50.755263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:50.755495 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.755411    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:16:50.755879 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:50.755730    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:50.755879 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:50.755845    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:16:51.755835 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:51.755522    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:51.755835 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:51.755651    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:16:52.755671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:52.755585    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:52.755838 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:52.755586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:52.755838 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:52.755713    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:16:52.755838 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:52.755800    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:16:53.754889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:53.754856    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:53.755075 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:53.754990    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:16:54.195535 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:54.195450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:54.195935 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.195634    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:54.195935 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.195715    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret podName:4ee12eab-d883-42ae-94cc-6618823fad3c nodeName:}" failed. No retries permitted until 2026-07-23 18:17:02.195694347 +0000 UTC m=+18.043384855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret") pod "global-pull-secret-syncer-vqch6" (UID: "4ee12eab-d883-42ae-94cc-6618823fad3c") : object "kube-system"/"original-pull-secret" not registered
Jul 23 18:16:54.296174 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:54.296043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:54.296388 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.296196    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:54.296388 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.296263    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs podName:4e9413cd-4d10-4609-a427-55fcc8474d38 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:02.296244241 +0000 UTC m=+18.143934748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs") pod "network-metrics-daemon-75jnc" (UID: "4e9413cd-4d10-4609-a427-55fcc8474d38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:16:54.397230 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:54.397191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:54.397455 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.397380    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 18:16:54.397533 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.397465    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 18:16:54.397533 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.397479    2561 projected.go:196] Error preparing data for projected volume kube-api-access-n7kn7 for pod openshift-network-diagnostics/network-check-target-kwpb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:54.397636 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.397581    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7 podName:a4abfb91-e90d-450f-8a07-b6fa81d7a6d0 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:02.397548811 +0000 UTC m=+18.245239331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7kn7" (UniqueName: "kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7") pod "network-check-target-kwpb4" (UID: "a4abfb91-e90d-450f-8a07-b6fa81d7a6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:16:54.756931 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:54.756349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:54.756931 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:54.756394    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:54.756931 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.756569    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:16:54.756931 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:54.756628    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:16:55.755184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:55.754815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:55.755184 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:55.754938    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:16:56.754620 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:56.754584    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:56.754620 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:56.754619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:56.754862 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:56.754720    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:16:56.754862 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:56.754804    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:16:57.754835 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:57.754793    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:57.755227 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:57.754912    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:16:58.754973 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:58.754930    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:16:58.755451 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:58.755076    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:16:58.755451 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:58.755134    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:16:58.755451 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:58.755241    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:16:59.755565 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:16:59.755534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:16:59.755962 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:16:59.755643    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:00.755617 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:00.755581    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:00.756004 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:00.755714    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:00.756004 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:00.755772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:00.756004 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:00.755886    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:01.755206 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:01.755169    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:01.755445 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:01.755301    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:02.257298 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:02.257261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:02.257767 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.257426    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 18:17:02.257767 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.257508    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret podName:4ee12eab-d883-42ae-94cc-6618823fad3c nodeName:}" failed. No retries permitted until 2026-07-23 18:17:18.257487035 +0000 UTC m=+34.105177541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret") pod "global-pull-secret-syncer-vqch6" (UID: "4ee12eab-d883-42ae-94cc-6618823fad3c") : object "kube-system"/"original-pull-secret" not registered
Jul 23 18:17:02.357703 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:02.357673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:02.357856 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.357823    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:17:02.357934 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.357880    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs podName:4e9413cd-4d10-4609-a427-55fcc8474d38 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:18.357864467 +0000 UTC m=+34.205554979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs") pod "network-metrics-daemon-75jnc" (UID: "4e9413cd-4d10-4609-a427-55fcc8474d38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:17:02.458177 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:02.458144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:02.458385 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.458365    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 18:17:02.458458 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.458388    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 18:17:02.458458 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.458406    2561 projected.go:196] Error preparing data for projected volume kube-api-access-n7kn7 for pod openshift-network-diagnostics/network-check-target-kwpb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:17:02.458561 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.458468    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7 podName:a4abfb91-e90d-450f-8a07-b6fa81d7a6d0 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:18.45844876 +0000 UTC m=+34.306139269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7kn7" (UniqueName: "kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7") pod "network-check-target-kwpb4" (UID: "a4abfb91-e90d-450f-8a07-b6fa81d7a6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:17:02.754868 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:02.754792    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:02.755023 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.754915    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:02.755023 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:02.754966    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:02.755127 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:02.755050    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:03.755029 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:03.754992    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:03.755438 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:03.755105    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:04.755872 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:04.755830    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:04.756365 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:04.755937    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:04.756365 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:04.755986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:04.756365 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:04.756106    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:05.755060 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:05.755021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:05.755219 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:05.755137    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:06.755632 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:06.755548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:06.755959 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:06.755658    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:06.755959 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:06.755685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:06.755959 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:06.755800    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:06.997055 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:06.996812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rk2rs" event={"ID":"40b7b2b4-947f-42bf-9f83-6bd57c73251e","Type":"ContainerStarted","Data":"dc898145c6cb27cf128a8b75386aa1bb73d0acb3937daaa3d8e5b1d2a259d43a"}
Jul 23 18:17:06.998623 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:06.998596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6gpl" event={"ID":"39bf1855-8430-4c11-a577-32be87bf9fa5","Type":"ContainerStarted","Data":"0b35ccc3467f299697f0fb93b1810a04a88d5421458964c47575dd0674962f87"}
Jul 23 18:17:07.000442 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.000100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal" event={"ID":"292e00e11710d5a1a45e5b6a5591eaca","Type":"ContainerStarted","Data":"1975c7c701eaf9a96bbe11ff3342876f1ace0e6217d7fce0e9e16947e4694ba5"}
Jul 23 18:17:07.001776 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.001740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5brsd" event={"ID":"23960491-7757-4a97-8f6f-445e6c5daac5","Type":"ContainerStarted","Data":"8c8cf527d791f74182e09122d9fac95c1c7bd1b17849bb866a45159f6a6e4757"}
Jul 23 18:17:07.003338 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.003314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn" event={"ID":"307a0a42-dfca-4ad1-a5e2-0afa874975a4","Type":"ContainerStarted","Data":"9d5ee332afcf525e0a7f31f711a526e176fb863b922b4a7a15fec21618b8c03b"}
Jul 23 18:17:07.004938 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.004912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d44gh" event={"ID":"8a8557ac-27ae-4f77-8c7a-1d32266f570e","Type":"ContainerStarted","Data":"2625120cd1c243d0a6b9d7f582cfee2a36d72727b182f6b44a850df6f61ac013"}
Jul 23 18:17:07.006995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.006772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v4g5k" event={"ID":"e804a5b4-b905-4966-a132-976a3d67e6dc","Type":"ContainerStarted","Data":"0bf1fe9b697295b6b529b80943aa954c9b9e20d52b233d7ee8e5a95902a44c36"}
Jul 23 18:17:07.008442 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.008421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h86v2" event={"ID":"abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb","Type":"ContainerStarted","Data":"e9b213cc4d3071e8c591e66fc9e7b5368ec836e62e1191447b472442f29f24fa"}
Jul 23 18:17:07.013084 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.013018    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rk2rs" podStartSLOduration=2.155585045 podStartE2EDuration="22.01299743s" totalImagesPullingTime="19.857412385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:07.012620018 +0000 UTC m=+22.860310743" watchObservedRunningTime="2026-07-23 18:17:07.01299743 +0000 UTC m=+22.860688020"
Jul 23 18:17:07.013183 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.013151    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-91.ec2.internal" podStartSLOduration=22.013141184 podStartE2EDuration="22.013141184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:49.982543109 +0000 UTC m=+5.830233642" watchObservedRunningTime="2026-07-23 18:17:07.013141184 +0000 UTC m=+22.860831713"
Jul 23 18:17:07.033610 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.031668    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5brsd" podStartSLOduration=2.205757477 podStartE2EDuration="22.031642488s" totalImagesPullingTime="19.825885011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:07.030947872 +0000 UTC m=+22.878638400" watchObservedRunningTime="2026-07-23 18:17:07.031642488 +0000 UTC m=+22.879333018"
Jul 23 18:17:07.098806 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.098761    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j6gpl" podStartSLOduration=2.236471069 podStartE2EDuration="22.098745275s" totalImagesPullingTime="19.862274206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:07.079250405 +0000 UTC m=+22.926940933" watchObservedRunningTime="2026-07-23 18:17:07.098745275 +0000 UTC m=+22.946435801"
Jul 23 18:17:07.098959 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.098908    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h86v2" podStartSLOduration=2.2155944 podStartE2EDuration="22.098901438s" totalImagesPullingTime="19.883307038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:07.098593226 +0000 UTC m=+22.946283752" watchObservedRunningTime="2026-07-23 18:17:07.098901438 +0000 UTC m=+22.946591964"
Jul 23 18:17:07.113131 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.113083    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v4g5k" podStartSLOduration=2.2942 podStartE2EDuration="22.113067229s" totalImagesPullingTime="19.818867229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:07.112593125 +0000 UTC m=+22.960283653" watchObservedRunningTime="2026-07-23 18:17:07.113067229 +0000 UTC m=+22.960757755"
Jul 23 18:17:07.126956 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.126913    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-91.ec2.internal" podStartSLOduration=22.126896204 podStartE2EDuration="22.126896204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:07.126516607 +0000 UTC m=+22.974207134" watchObservedRunningTime="2026-07-23 18:17:07.126896204 +0000 UTC m=+22.974586734"
Jul 23 18:17:07.755557 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.755373    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:07.755750 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:07.755659    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:07.979534 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.979504    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:17:07.980377 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:07.980358    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:17:08.011714 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.011681    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a8557ac-27ae-4f77-8c7a-1d32266f570e" containerID="2625120cd1c243d0a6b9d7f582cfee2a36d72727b182f6b44a850df6f61ac013" exitCode=0
Jul 23 18:17:08.011853 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.011747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d44gh" event={"ID":"8a8557ac-27ae-4f77-8c7a-1d32266f570e","Type":"ContainerDied","Data":"2625120cd1c243d0a6b9d7f582cfee2a36d72727b182f6b44a850df6f61ac013"}
Jul 23 18:17:08.013163 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.013141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wr5zm" event={"ID":"dcad1db1-cdf1-465a-bb2b-05dfda66096e","Type":"ContainerStarted","Data":"07262442eef302aa8e74538cfab9669c565c9f1608609e2915ae9297ff63cc33"}
Jul 23 18:17:08.015824 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.015805    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:17:08.016171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.016149    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9052287-a150-41b0-b0ef-82313217f2e7" containerID="46e68f8a941c558dd906d016e99368e456699003c6dde130f18718882048f08a" exitCode=1
Jul 23 18:17:08.016272 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.016246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerStarted","Data":"6cd9fa950409662bc0f2ef454f2cf63cf9283e8e233aae3450ad2a19aee39e49"}
Jul 23 18:17:08.016396 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.016341    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerStarted","Data":"5e95bd2d806926a9f5717c9157b0292cfa3c3a502ea09480eb6b5eaa732a2df5"}
Jul 23 18:17:08.016396 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.016393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerStarted","Data":"eb1e42d580c4c98918dde1bbb90be8cd670cb9820aa8acb7cf168b73867874cc"}
Jul 23 18:17:08.016508 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.016440    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerStarted","Data":"4a4a047ff73767e87ca55b5e51adbee961b0f683dad783865b50efa2006b71da"}
Jul 23 18:17:08.016508 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.016494    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:17:08.016608 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.016514    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerDied","Data":"46e68f8a941c558dd906d016e99368e456699003c6dde130f18718882048f08a"}
Jul 23 18:17:08.016608 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.016564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerStarted","Data":"62c902113b3c0ab487f635491424ab01fca9e03abf6ef152a97edf1783960da9"}
Jul 23 18:17:08.017060 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.017044    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j6gpl"
Jul 23 18:17:08.156489 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.156466    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 18:17:08.600598 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.600488    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T18:17:08.15648496Z","UUID":"e300242c-9fdd-4cbe-a5a0-ae7bf2657d7a","Handler":null,"Name":"","Endpoint":""}
Jul 23 18:17:08.601989 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.601970    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 18:17:08.602107 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.602008    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 18:17:08.755431 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.755404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:08.755613 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:08.755519    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:08.755613 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:08.755578    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:08.755728 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:08.755678    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:09.021301 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:09.021182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn" event={"ID":"307a0a42-dfca-4ad1-a5e2-0afa874975a4","Type":"ContainerStarted","Data":"88053f5befe1a11691072382b0be32d4361e8c45c0033d4514c4307821b2236e"}
Jul 23 18:17:09.754616 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:09.754587    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:09.754809 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:09.754691    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:10.026958 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:10.026920    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn" event={"ID":"307a0a42-dfca-4ad1-a5e2-0afa874975a4","Type":"ContainerStarted","Data":"acce2bdbb68bcce4d8481fd5cc3774c3cecbfca561b374c8a601afd6e6b0c594"}
Jul 23 18:17:10.046107 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:10.046065    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wr5zm" podStartSLOduration=5.202834971 podStartE2EDuration="25.046049622s" totalImagesPullingTime="19.843214651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:08.062946986 +0000 UTC m=+23.910637513" watchObservedRunningTime="2026-07-23 18:17:10.046049622 +0000 UTC m=+25.893740150"
Jul 23 18:17:10.046235 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:10.046138    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wscnn" podStartSLOduration=3.056748077 podStartE2EDuration="25.046133736s" totalImagesPullingTime="21.989385659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:10.045749536 +0000 UTC m=+25.893440103" watchObservedRunningTime="2026-07-23 18:17:10.046133736 +0000 UTC m=+25.893824263"
Jul 23 18:17:10.755051 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:10.754874    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:10.755233 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:10.754893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:10.755233 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:10.755168    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:10.755233 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:10.755219    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:11.032341 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:11.032268    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:17:11.032762 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:11.032726    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerStarted","Data":"7f42d09f476747d82ea4e3efad6829e6a71c627e206672122eca8af6a7d07203"}
Jul 23 18:17:11.755151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:11.755122    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:11.755350 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:11.755243    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:12.755211 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:12.755023    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:12.755798 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:12.755023    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:12.755798 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:12.755330    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:12.755798 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:12.755407    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:13.038459 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:13.038383    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a8557ac-27ae-4f77-8c7a-1d32266f570e" containerID="45455fe5e2a1bd454a16c4c02d6639c2db5e8987e28be07667a6c10cb4a02dd5" exitCode=0
Jul 23 18:17:13.038606 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:13.038464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d44gh" event={"ID":"8a8557ac-27ae-4f77-8c7a-1d32266f570e","Type":"ContainerDied","Data":"45455fe5e2a1bd454a16c4c02d6639c2db5e8987e28be07667a6c10cb4a02dd5"}
Jul 23 18:17:13.041753 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:13.041737    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:17:13.042100 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:13.042082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerStarted","Data":"e646a0d7de6bf364f2909fcf3aecabb6d06e04e19734378bd7822b7094ac7862"}
Jul 23 18:17:13.042359 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:13.042342    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:17:13.042443 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:13.042413    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:17:13.042489 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:13.042482    2561 scope.go:117] "RemoveContainer" containerID="46e68f8a941c558dd906d016e99368e456699003c6dde130f18718882048f08a"
Jul 23 18:17:13.058433 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:13.058412    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:17:13.755071 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:13.755039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:13.755301 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:13.755138    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:14.047349 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:14.047274    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:17:14.047594 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:14.047573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" event={"ID":"d9052287-a150-41b0-b0ef-82313217f2e7","Type":"ContainerStarted","Data":"e0915543fc1989da116de62dcf5457bb006b3bff17cbc3cdafb6a6f835c4d994"}
Jul 23 18:17:14.047759 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:14.047743    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:17:14.062048 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:14.062025    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:17:14.076276 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:14.076235    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn" podStartSLOduration=8.863417289000001 podStartE2EDuration="29.076222359s" totalImagesPullingTime="20.21280507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:14.074891695 +0000 UTC m=+29.922582232" watchObservedRunningTime="2026-07-23 18:17:14.076222359 +0000 UTC m=+29.923912888"
Jul 23 18:17:14.755727 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:14.755697    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:14.756273 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:14.755804    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:14.756273 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:14.755853    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:14.756273 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:14.755904    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:15.051376 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:15.051280    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a8557ac-27ae-4f77-8c7a-1d32266f570e" containerID="711975e01a7fb827303a5a6a600896183509f62a3418df6e5d54b295f50380a7" exitCode=0
Jul 23 18:17:15.051376 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:15.051342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d44gh" event={"ID":"8a8557ac-27ae-4f77-8c7a-1d32266f570e","Type":"ContainerDied","Data":"711975e01a7fb827303a5a6a600896183509f62a3418df6e5d54b295f50380a7"}
Jul 23 18:17:15.754789 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:15.754760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:15.754947 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:15.754890    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:16.755544 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:16.755511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:16.755909 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:16.755510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:16.755909 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:16.755653    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:16.755909 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:16.755696    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:17.060543 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:17.060460    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a8557ac-27ae-4f77-8c7a-1d32266f570e" containerID="557dc76416b76129c1388c490ad5cf89fd4dee0a6554c81c9038a9db74bbea6a" exitCode=0
Jul 23 18:17:17.060543 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:17.060501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d44gh" event={"ID":"8a8557ac-27ae-4f77-8c7a-1d32266f570e","Type":"ContainerDied","Data":"557dc76416b76129c1388c490ad5cf89fd4dee0a6554c81c9038a9db74bbea6a"}
Jul 23 18:17:17.754788 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:17.754756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:17.754955 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:17.754853    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:18.273699 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:18.273663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:18.274148 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.273795    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 18:17:18.274148 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.273883    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret podName:4ee12eab-d883-42ae-94cc-6618823fad3c nodeName:}" failed. No retries permitted until 2026-07-23 18:17:50.273859916 +0000 UTC m=+66.121550425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret") pod "global-pull-secret-syncer-vqch6" (UID: "4ee12eab-d883-42ae-94cc-6618823fad3c") : object "kube-system"/"original-pull-secret" not registered
Jul 23 18:17:18.374580 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:18.374544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:18.374754 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.374661    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:17:18.374754 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.374730    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs podName:4e9413cd-4d10-4609-a427-55fcc8474d38 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:50.374711316 +0000 UTC m=+66.222401822 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs") pod "network-metrics-daemon-75jnc" (UID: "4e9413cd-4d10-4609-a427-55fcc8474d38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 18:17:18.474851 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:18.474811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:18.475032 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.475000    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 18:17:18.475032 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.475025    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 18:17:18.475124 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.475038    2561 projected.go:196] Error preparing data for projected volume kube-api-access-n7kn7 for pod openshift-network-diagnostics/network-check-target-kwpb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:17:18.475124 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.475102    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7 podName:a4abfb91-e90d-450f-8a07-b6fa81d7a6d0 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:50.475084174 +0000 UTC m=+66.322774697 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7kn7" (UniqueName: "kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7") pod "network-check-target-kwpb4" (UID: "a4abfb91-e90d-450f-8a07-b6fa81d7a6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 18:17:18.754831 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:18.754787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:18.754831 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:18.754826    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:18.755101 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.754934    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:18.755101 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:18.755081    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:19.755483 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:19.755453    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:19.755900 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:19.755566    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:20.755564 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:20.755528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:20.755978 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:20.755676    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:20.755978 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:20.755735    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:20.755978 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:20.755828    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:21.755100 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:21.755072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:21.755260 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:21.755167    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:22.075086 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:22.075004    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a8557ac-27ae-4f77-8c7a-1d32266f570e" containerID="e94464944d4f6b25127946cbf4d2882ac01bdde4a34982d96ae3568d475bdeca" exitCode=0
Jul 23 18:17:22.075086 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:22.075062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d44gh" event={"ID":"8a8557ac-27ae-4f77-8c7a-1d32266f570e","Type":"ContainerDied","Data":"e94464944d4f6b25127946cbf4d2882ac01bdde4a34982d96ae3568d475bdeca"}
Jul 23 18:17:22.754816 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:22.754792    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:22.754983 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:22.754888    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:22.754983 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:22.754964    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:22.755110 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:22.755058    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:23.079178 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:23.079099    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a8557ac-27ae-4f77-8c7a-1d32266f570e" containerID="d6434d6cd1e14d1a4a57af1b084f2409a5fd401ce820f40d65c8b2148c0ff1d5" exitCode=0
Jul 23 18:17:23.079178 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:23.079161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d44gh" event={"ID":"8a8557ac-27ae-4f77-8c7a-1d32266f570e","Type":"ContainerDied","Data":"d6434d6cd1e14d1a4a57af1b084f2409a5fd401ce820f40d65c8b2148c0ff1d5"}
Jul 23 18:17:23.755008 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:23.754974    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:23.755169 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:23.755073    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:24.084699 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:24.084483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d44gh" event={"ID":"8a8557ac-27ae-4f77-8c7a-1d32266f570e","Type":"ContainerStarted","Data":"8f2988edc31b95289b57412e6b6d0b79c16bec5dfeec939da3711bcf4b1ce066"}
Jul 23 18:17:24.111463 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:24.111403    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d44gh" podStartSLOduration=8.207840281 podStartE2EDuration="39.111385546s" totalImagesPullingTime="30.903545265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:24.110837531 +0000 UTC m=+39.958528080" watchObservedRunningTime="2026-07-23 18:17:24.111385546 +0000 UTC m=+39.959076073"
Jul 23 18:17:24.451529 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:24.451499    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vqch6"]
Jul 23 18:17:24.451693 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:24.451611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:24.451769 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:24.451705    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:24.454736 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:24.454681    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-75jnc"]
Jul 23 18:17:24.454858 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:24.454828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:24.454958 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:24.454938    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:24.455443 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:24.455420    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kwpb4"]
Jul 23 18:17:24.455528 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:24.455504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:24.455600 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:24.455581    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:25.755409 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:25.755378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:25.755998 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:25.755379    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:25.755998 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:25.755473    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwpb4" podUID="a4abfb91-e90d-450f-8a07-b6fa81d7a6d0"
Jul 23 18:17:25.755998 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:25.755553    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-75jnc" podUID="4e9413cd-4d10-4609-a427-55fcc8474d38"
Jul 23 18:17:26.755027 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:26.754997    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:26.755190 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:26.755126    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqch6" podUID="4ee12eab-d883-42ae-94cc-6618823fad3c"
Jul 23 18:17:27.424510 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.424485    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-91.ec2.internal" event="NodeReady"
Jul 23 18:17:27.424906 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.424602    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 18:17:27.468588 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.468557    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gg4kt"]
Jul 23 18:17:27.484921 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.484895    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b87z7"]
Jul 23 18:17:27.485051 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.485023    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.487241 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.487202    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w4ck5\""
Jul 23 18:17:27.487391 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.487350    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 18:17:27.487707 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.487694    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 18:17:27.491547 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.491527    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gg4kt"]
Jul 23 18:17:27.491684 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.491670    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b87z7"]
Jul 23 18:17:27.491758 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.491611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b87z7"
Jul 23 18:17:27.493910 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.493888    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zbtqj\""
Jul 23 18:17:27.494013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.493934    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 18:17:27.494013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.493942    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 18:17:27.494013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.493899    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 18:17:27.572145 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.572063    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7p9nd"]
Jul 23 18:17:27.588254 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.588228    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7p9nd"]
Jul 23 18:17:27.588392 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.588347    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.591118 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.591078    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 18:17:27.591118 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.591092    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 18:17:27.591296 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.591139    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 18:17:27.591296 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.591148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ccngx\""
Jul 23 18:17:27.591296 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.591155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 18:17:27.645633 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.645607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-tmp-dir\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.645633 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.645635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggr4q\" (UniqueName: \"kubernetes.io/projected/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-kube-api-access-ggr4q\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.645798 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.645672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-config-volume\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.645798 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.645693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a77fc19-bdd8-44c3-afc3-b03e0954adc7-cert\") pod \"ingress-canary-b87z7\" (UID: \"1a77fc19-bdd8-44c3-afc3-b03e0954adc7\") " pod="openshift-ingress-canary/ingress-canary-b87z7"
Jul 23 18:17:27.645798 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.645727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmr5m\" (UniqueName: \"kubernetes.io/projected/1a77fc19-bdd8-44c3-afc3-b03e0954adc7-kube-api-access-kmr5m\") pod \"ingress-canary-b87z7\" (UID: \"1a77fc19-bdd8-44c3-afc3-b03e0954adc7\") " pod="openshift-ingress-canary/ingress-canary-b87z7"
Jul 23 18:17:27.645798 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.645760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-metrics-tls\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.746949 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.746918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-metrics-tls\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.747101 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.746956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee48386e-15b8-49c1-b429-eaa71ba01cbd-data-volume\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.747101 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.746978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee48386e-15b8-49c1-b429-eaa71ba01cbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.747101 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.746997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-tmp-dir\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.747101 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.747083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggr4q\" (UniqueName: \"kubernetes.io/projected/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-kube-api-access-ggr4q\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.747364 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.747119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee48386e-15b8-49c1-b429-eaa71ba01cbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.747364 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.747178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-config-volume\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.747364 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.747204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a77fc19-bdd8-44c3-afc3-b03e0954adc7-cert\") pod \"ingress-canary-b87z7\" (UID: \"1a77fc19-bdd8-44c3-afc3-b03e0954adc7\") " pod="openshift-ingress-canary/ingress-canary-b87z7"
Jul 23 18:17:27.747364 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.747254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee48386e-15b8-49c1-b429-eaa71ba01cbd-crio-socket\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.747364 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.747325    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzg7\" (UniqueName: \"kubernetes.io/projected/ee48386e-15b8-49c1-b429-eaa71ba01cbd-kube-api-access-bhzg7\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.747604 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.747407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmr5m\" (UniqueName: \"kubernetes.io/projected/1a77fc19-bdd8-44c3-afc3-b03e0954adc7-kube-api-access-kmr5m\") pod \"ingress-canary-b87z7\" (UID: \"1a77fc19-bdd8-44c3-afc3-b03e0954adc7\") " pod="openshift-ingress-canary/ingress-canary-b87z7"
Jul 23 18:17:27.747604 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.747415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-tmp-dir\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.749699 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.748037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-config-volume\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.753441 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.753414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-metrics-tls\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.753559 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.753508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a77fc19-bdd8-44c3-afc3-b03e0954adc7-cert\") pod \"ingress-canary-b87z7\" (UID: \"1a77fc19-bdd8-44c3-afc3-b03e0954adc7\") " pod="openshift-ingress-canary/ingress-canary-b87z7"
Jul 23 18:17:27.754592 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.754572    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:27.754662 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.754609    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:27.754780 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.754759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggr4q\" (UniqueName: \"kubernetes.io/projected/906c127e-72e7-4a3d-888d-7e0a5aa0cd9c-kube-api-access-ggr4q\") pod \"dns-default-gg4kt\" (UID: \"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c\") " pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.755361 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.755344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmr5m\" (UniqueName: \"kubernetes.io/projected/1a77fc19-bdd8-44c3-afc3-b03e0954adc7-kube-api-access-kmr5m\") pod \"ingress-canary-b87z7\" (UID: \"1a77fc19-bdd8-44c3-afc3-b03e0954adc7\") " pod="openshift-ingress-canary/ingress-canary-b87z7"
Jul 23 18:17:27.756851 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.756833    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 18:17:27.756931 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.756883    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsfk8\""
Jul 23 18:17:27.756931 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.756841    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 18:17:27.756931 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.756833    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 18:17:27.757055 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.756938    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w7wd2\""
Jul 23 18:17:27.796085 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.796063    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:27.802180 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.802158    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b87z7"
Jul 23 18:17:27.848079 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.847928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee48386e-15b8-49c1-b429-eaa71ba01cbd-data-volume\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.848190 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.848097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee48386e-15b8-49c1-b429-eaa71ba01cbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.848190 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.848124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee48386e-15b8-49c1-b429-eaa71ba01cbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.848190 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.848164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee48386e-15b8-49c1-b429-eaa71ba01cbd-crio-socket\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.848190 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.848180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzg7\" (UniqueName: \"kubernetes.io/projected/ee48386e-15b8-49c1-b429-eaa71ba01cbd-kube-api-access-bhzg7\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.848375 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.848222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee48386e-15b8-49c1-b429-eaa71ba01cbd-data-volume\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.848406 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.848378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ee48386e-15b8-49c1-b429-eaa71ba01cbd-crio-socket\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.848830 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.848805    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ee48386e-15b8-49c1-b429-eaa71ba01cbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.851087 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.851041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ee48386e-15b8-49c1-b429-eaa71ba01cbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.856551 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.856529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzg7\" (UniqueName: \"kubernetes.io/projected/ee48386e-15b8-49c1-b429-eaa71ba01cbd-kube-api-access-bhzg7\") pod \"insights-runtime-extractor-7p9nd\" (UID: \"ee48386e-15b8-49c1-b429-eaa71ba01cbd\") " pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:27.896569 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:27.896540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7p9nd"
Jul 23 18:17:28.012330 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:28.012298    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b87z7"]
Jul 23 18:17:28.016113 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:28.016082    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gg4kt"]
Jul 23 18:17:28.016673 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:28.016646    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a77fc19_bdd8_44c3_afc3_b03e0954adc7.slice/crio-09e63fa3b6cade1a6645c7e97b4a99018cdbf5d771b33401a685779b7932347f WatchSource:0}: Error finding container 09e63fa3b6cade1a6645c7e97b4a99018cdbf5d771b33401a685779b7932347f: Status 404 returned error can't find the container with id 09e63fa3b6cade1a6645c7e97b4a99018cdbf5d771b33401a685779b7932347f
Jul 23 18:17:28.019343 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:28.019324    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906c127e_72e7_4a3d_888d_7e0a5aa0cd9c.slice/crio-dae2944124de630ef78e4b39f1c1c84932b3a598a02be90dd08eb2f6d3ce8773 WatchSource:0}: Error finding container dae2944124de630ef78e4b39f1c1c84932b3a598a02be90dd08eb2f6d3ce8773: Status 404 returned error can't find the container with id dae2944124de630ef78e4b39f1c1c84932b3a598a02be90dd08eb2f6d3ce8773
Jul 23 18:17:28.050552 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:28.050525    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7p9nd"]
Jul 23 18:17:28.055089 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:28.055065    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee48386e_15b8_49c1_b429_eaa71ba01cbd.slice/crio-e17c7ce4cb0ea5e327bc8ea694ea914bd7e516d5f102b7e19026a858e082b08c WatchSource:0}: Error finding container e17c7ce4cb0ea5e327bc8ea694ea914bd7e516d5f102b7e19026a858e082b08c: Status 404 returned error can't find the container with id e17c7ce4cb0ea5e327bc8ea694ea914bd7e516d5f102b7e19026a858e082b08c
Jul 23 18:17:28.095080 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:28.094967    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b87z7" event={"ID":"1a77fc19-bdd8-44c3-afc3-b03e0954adc7","Type":"ContainerStarted","Data":"09e63fa3b6cade1a6645c7e97b4a99018cdbf5d771b33401a685779b7932347f"}
Jul 23 18:17:28.096083 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:28.096062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gg4kt" event={"ID":"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c","Type":"ContainerStarted","Data":"dae2944124de630ef78e4b39f1c1c84932b3a598a02be90dd08eb2f6d3ce8773"}
Jul 23 18:17:28.097049 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:28.097029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7p9nd" event={"ID":"ee48386e-15b8-49c1-b429-eaa71ba01cbd","Type":"ContainerStarted","Data":"e17c7ce4cb0ea5e327bc8ea694ea914bd7e516d5f102b7e19026a858e082b08c"}
Jul 23 18:17:28.754942 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:28.754905    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:28.757731 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:28.757713    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 18:17:31.433504 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:31.433469    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"]
Jul 23 18:17:31.472631 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:31.472602    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"]
Jul 23 18:17:31.472801 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:31.472702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"
Jul 23 18:17:31.475087 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:31.474955    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 23 18:17:31.475087 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:31.475051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hljck\""
Jul 23 18:17:31.576915 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:31.576883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e492398-f467-4e95-9138-3dc099fd9943-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-c9qkz\" (UID: \"9e492398-f467-4e95-9138-3dc099fd9943\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"
Jul 23 18:17:31.678316 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:31.678219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e492398-f467-4e95-9138-3dc099fd9943-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-c9qkz\" (UID: \"9e492398-f467-4e95-9138-3dc099fd9943\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"
Jul 23 18:17:31.681843 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:31.681814    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e492398-f467-4e95-9138-3dc099fd9943-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-c9qkz\" (UID: \"9e492398-f467-4e95-9138-3dc099fd9943\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"
Jul 23 18:17:31.783370 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:31.783330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"
Jul 23 18:17:32.504529 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.504493    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b85b9bbb7-f2vqq"]
Jul 23 18:17:32.603421 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.603375    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b85b9bbb7-f2vqq"]
Jul 23 18:17:32.603603 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.603471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.606435 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.606407    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 23 18:17:32.607173 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.607115    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 23 18:17:32.607173 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.607158    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 23 18:17:32.607369 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.607310    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 23 18:17:32.607369 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.607342    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 23 18:17:32.607472 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.607408    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 23 18:17:32.607472 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.607426    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fnxx6\""
Jul 23 18:17:32.608045 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.607978    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 23 18:17:32.612007 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.611752    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 23 18:17:32.786994 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.786915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-trusted-ca-bundle\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.786994 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.786965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlxt6\" (UniqueName: \"kubernetes.io/projected/d3bd374a-43b1-45f7-bb42-39c0652ed26e-kube-api-access-wlxt6\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.787212 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.786998    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-config\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.787212 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.787030    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-service-ca\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.787212 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.787058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-oauth-serving-cert\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.787212 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.787080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-oauth-config\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.787212 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.787115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-serving-cert\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.888600 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.888557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-trusted-ca-bundle\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.888776 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.888606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlxt6\" (UniqueName: \"kubernetes.io/projected/d3bd374a-43b1-45f7-bb42-39c0652ed26e-kube-api-access-wlxt6\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.888776 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.888644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-config\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.888776 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.888681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-service-ca\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.888776 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.888725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-oauth-serving-cert\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.888776 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.888757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-oauth-config\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.889023 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.888792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-serving-cert\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.889553 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.889516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-trusted-ca-bundle\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.889831 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.889805    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-service-ca\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.889973 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.889929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-oauth-serving-cert\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.890074 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.889975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-config\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.892097 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.892065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-oauth-config\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.892184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.892155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-serving-cert\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.897961 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.897940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlxt6\" (UniqueName: \"kubernetes.io/projected/d3bd374a-43b1-45f7-bb42-39c0652ed26e-kube-api-access-wlxt6\") pod \"console-b85b9bbb7-f2vqq\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") " pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:32.915328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:32.915300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:34.688634 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:34.688351    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"]
Jul 23 18:17:34.698487 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:34.698453    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b85b9bbb7-f2vqq"]
Jul 23 18:17:34.761355 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:34.761315    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e492398_f467_4e95_9138_3dc099fd9943.slice/crio-b76fcb4ae8adec7316b1460a1a3d0691ee454a67d1256685334fe3061819aa6e WatchSource:0}: Error finding container b76fcb4ae8adec7316b1460a1a3d0691ee454a67d1256685334fe3061819aa6e: Status 404 returned error can't find the container with id b76fcb4ae8adec7316b1460a1a3d0691ee454a67d1256685334fe3061819aa6e
Jul 23 18:17:34.762299 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:34.762251    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bd374a_43b1_45f7_bb42_39c0652ed26e.slice/crio-7cc3898e1b05fd9a815512a5e7b81643884fd825844f136536523651cb8ad6bc WatchSource:0}: Error finding container 7cc3898e1b05fd9a815512a5e7b81643884fd825844f136536523651cb8ad6bc: Status 404 returned error can't find the container with id 7cc3898e1b05fd9a815512a5e7b81643884fd825844f136536523651cb8ad6bc
Jul 23 18:17:35.118711 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:35.118596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7p9nd" event={"ID":"ee48386e-15b8-49c1-b429-eaa71ba01cbd","Type":"ContainerStarted","Data":"6c62386d0de5072e7f255b10ca4891a8d8d059e2a5ebae756c1a157d9c2f8b9b"}
Jul 23 18:17:35.119728 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:35.119705    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz" event={"ID":"9e492398-f467-4e95-9138-3dc099fd9943","Type":"ContainerStarted","Data":"b76fcb4ae8adec7316b1460a1a3d0691ee454a67d1256685334fe3061819aa6e"}
Jul 23 18:17:35.121107 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:35.121082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b87z7" event={"ID":"1a77fc19-bdd8-44c3-afc3-b03e0954adc7","Type":"ContainerStarted","Data":"015b757eb49bee239945457264778fa2ce3d6dba339b8bd5ef74d400fc1455a4"}
Jul 23 18:17:35.122819 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:35.122799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gg4kt" event={"ID":"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c","Type":"ContainerStarted","Data":"adb682eda1e983ec558e010154e5eafe11972180e74d178ac282973448c6fc7c"}
Jul 23 18:17:35.122913 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:35.122840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gg4kt" event={"ID":"906c127e-72e7-4a3d-888d-7e0a5aa0cd9c","Type":"ContainerStarted","Data":"99da5abd5581ff275c5108fbbb9e2b49a5a17bd8ec906d023875d57f6de55392"}
Jul 23 18:17:35.122913 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:35.122861    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:35.123861 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:35.123843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b85b9bbb7-f2vqq" event={"ID":"d3bd374a-43b1-45f7-bb42-39c0652ed26e","Type":"ContainerStarted","Data":"7cc3898e1b05fd9a815512a5e7b81643884fd825844f136536523651cb8ad6bc"}
Jul 23 18:17:35.137904 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:35.137851    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b87z7" podStartSLOduration=1.637078204 podStartE2EDuration="8.137836583s" totalImagesPullingTime="6.500758379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:35.136684157 +0000 UTC m=+50.984374685" watchObservedRunningTime="2026-07-23 18:17:35.137836583 +0000 UTC m=+50.985527112"
Jul 23 18:17:35.155108 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:35.155044    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gg4kt" podStartSLOduration=1.660200089 podStartE2EDuration="8.155033154s" totalImagesPullingTime="6.494833065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:35.15415495 +0000 UTC m=+51.001845477" watchObservedRunningTime="2026-07-23 18:17:35.155033154 +0000 UTC m=+51.002723707"
Jul 23 18:17:41.147256 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.147210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b85b9bbb7-f2vqq" event={"ID":"d3bd374a-43b1-45f7-bb42-39c0652ed26e","Type":"ContainerStarted","Data":"e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870"}
Jul 23 18:17:41.149092 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.149057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7p9nd" event={"ID":"ee48386e-15b8-49c1-b429-eaa71ba01cbd","Type":"ContainerStarted","Data":"4cfdcc7f56d91767f825c4913c45f9e6dd495981aa10577b8df22e0f4bcaea7f"}
Jul 23 18:17:41.150610 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.150586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz" event={"ID":"9e492398-f467-4e95-9138-3dc099fd9943","Type":"ContainerStarted","Data":"e429537b984ccf0089e5759e0fcc0f9013a9906584dfd3f257cde1531a0f6185"}
Jul 23 18:17:41.150822 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.150798    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"
Jul 23 18:17:41.156684 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.156656    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz"
Jul 23 18:17:41.167788 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.167746    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b85b9bbb7-f2vqq" podStartSLOduration=3.433349821 podStartE2EDuration="9.167733231s" totalImagesPullingTime="5.73438341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:41.166494157 +0000 UTC m=+57.014184686" watchObservedRunningTime="2026-07-23 18:17:41.167733231 +0000 UTC m=+57.015423752"
Jul 23 18:17:41.183066 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.183017    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-c9qkz" podStartSLOduration=4.470323189 podStartE2EDuration="10.182997052s" totalImagesPullingTime="5.712673863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:41.181770134 +0000 UTC m=+57.029460661" watchObservedRunningTime="2026-07-23 18:17:41.182997052 +0000 UTC m=+57.030687581"
Jul 23 18:17:41.505598 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.505565    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"]
Jul 23 18:17:41.521001 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.520966    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"]
Jul 23 18:17:41.521177 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.521087    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.524402 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.524374    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 23 18:17:41.524545 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.524405    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 18:17:41.524545 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.524439    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 18:17:41.524666 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.524601    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 18:17:41.524666 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.524615    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dcq46\""
Jul 23 18:17:41.524666 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.524624    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 23 18:17:41.565013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.564987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc0befc0-78e3-47f1-b522-826b27494639-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.565182 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.565024    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc0befc0-78e3-47f1-b522-826b27494639-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.565182 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.565135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nss72\" (UniqueName: \"kubernetes.io/projected/bc0befc0-78e3-47f1-b522-826b27494639-kube-api-access-nss72\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.565328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.565205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0befc0-78e3-47f1-b522-826b27494639-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.665711 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.665677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc0befc0-78e3-47f1-b522-826b27494639-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.665711 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.665727    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc0befc0-78e3-47f1-b522-826b27494639-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.665961 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.665789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nss72\" (UniqueName: \"kubernetes.io/projected/bc0befc0-78e3-47f1-b522-826b27494639-kube-api-access-nss72\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.665961 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.665842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0befc0-78e3-47f1-b522-826b27494639-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.666554 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.666526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc0befc0-78e3-47f1-b522-826b27494639-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.669963 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.669940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc0befc0-78e3-47f1-b522-826b27494639-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.669963 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.669954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc0befc0-78e3-47f1-b522-826b27494639-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.674920 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.674895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nss72\" (UniqueName: \"kubernetes.io/projected/bc0befc0-78e3-47f1-b522-826b27494639-kube-api-access-nss72\") pod \"prometheus-operator-5945b5cfd-lcxlm\" (UID: \"bc0befc0-78e3-47f1-b522-826b27494639\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.833320 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.833224    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"
Jul 23 18:17:41.991575 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:41.991541    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm"]
Jul 23 18:17:41.995403 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:41.995374    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0befc0_78e3_47f1_b522_826b27494639.slice/crio-f01a76bf5b61e86bbf5db68191374f22b6f517abab2d05ac79d92abd295b70e2 WatchSource:0}: Error finding container f01a76bf5b61e86bbf5db68191374f22b6f517abab2d05ac79d92abd295b70e2: Status 404 returned error can't find the container with id f01a76bf5b61e86bbf5db68191374f22b6f517abab2d05ac79d92abd295b70e2
Jul 23 18:17:42.155639 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:42.155553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm" event={"ID":"bc0befc0-78e3-47f1-b522-826b27494639","Type":"ContainerStarted","Data":"f01a76bf5b61e86bbf5db68191374f22b6f517abab2d05ac79d92abd295b70e2"}
Jul 23 18:17:42.915771 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:42.915737    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:42.915974 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:42.915938    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:42.921028 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:42.921002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:43.163622 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:43.163595    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:17:45.130347 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:45.130314    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gg4kt"
Jul 23 18:17:45.168878 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:45.168844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7p9nd" event={"ID":"ee48386e-15b8-49c1-b429-eaa71ba01cbd","Type":"ContainerStarted","Data":"b22de6d743db9662cbe38a903678173c3ff6b22a5f081db2b628820d30018773"}
Jul 23 18:17:45.219063 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:45.218999    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7p9nd" podStartSLOduration=1.9670923409999999 podStartE2EDuration="18.218981212s" totalImagesPullingTime="16.251888871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:17:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:45.217727866 +0000 UTC m=+61.065418394" watchObservedRunningTime="2026-07-23 18:17:45.218981212 +0000 UTC m=+61.066671739"
Jul 23 18:17:46.064170 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:46.064144    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm4gn"
Jul 23 18:17:46.173826 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:46.173781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm" event={"ID":"bc0befc0-78e3-47f1-b522-826b27494639","Type":"ContainerStarted","Data":"5b0b8d5ecdff830bb006fc455faaafc3bf2e2d36dcba3456e861893bfef93d2c"}
Jul 23 18:17:46.174235 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:46.173840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm" event={"ID":"bc0befc0-78e3-47f1-b522-826b27494639","Type":"ContainerStarted","Data":"f366a4ca7f5a6cd8496abf2e85dd4fd8551f4c5b063deb97cd1d2f40a1ced01c"}
Jul 23 18:17:46.197056 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:46.197012    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lcxlm" podStartSLOduration=1.780038 podStartE2EDuration="5.196997432s" totalImagesPullingTime="3.416959432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:46.195699081 +0000 UTC m=+62.043389617" watchObservedRunningTime="2026-07-23 18:17:46.196997432 +0000 UTC m=+62.044687960"
Jul 23 18:17:47.861179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.861130    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"]
Jul 23 18:17:47.864008 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.863988    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z5q82"]
Jul 23 18:17:47.864134 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.864112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:47.868436 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.867365    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 23 18:17:47.868436 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.867615    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qkcrc\""
Jul 23 18:17:47.868436 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.867845    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 23 18:17:47.869461 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.869436    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.871826 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.871803    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bvvxz\""
Jul 23 18:17:47.871969 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.871952    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 18:17:47.872435 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.872418    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 18:17:47.872830 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.872812    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 18:17:47.883814 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.883794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"]
Jul 23 18:17:47.898581 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.898561    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"]
Jul 23 18:17:47.900821 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.900807    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:47.903458 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.903437    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 23 18:17:47.903565 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.903468    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 23 18:17:47.903623 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.903602    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fdlt6\""
Jul 23 18:17:47.903750 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.903732    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 23 18:17:47.914005 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.913947    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"]
Jul 23 18:17:47.921483 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:47.921590 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921515    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-wtmp\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.921590 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9227e00f-e0cf-4a30-b287-1c645cf59134-metrics-client-ca\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.921590 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-tls\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.921590 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-textfile\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.921738 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:47.921738 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.921738 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtfh\" (UniqueName: \"kubernetes.io/projected/9227e00f-e0cf-4a30-b287-1c645cf59134-kube-api-access-mhtfh\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.921871 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9227e00f-e0cf-4a30-b287-1c645cf59134-sys\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.921871 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:47.921871 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9227e00f-e0cf-4a30-b287-1c645cf59134-root\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.922006 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-accelerators-collector-config\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:47.922006 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:47.921965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6jh6\" (UniqueName: \"kubernetes.io/projected/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-kube-api-access-j6jh6\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.023053 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-wtmp\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023053 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9227e00f-e0cf-4a30-b287-1c645cf59134-metrics-client-ca\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023321 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-tls\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023321 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-textfile\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023321 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.023321 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.023321 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023210    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-wtmp\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023321 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023321 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtfh\" (UniqueName: \"kubernetes.io/projected/9227e00f-e0cf-4a30-b287-1c645cf59134-kube-api-access-mhtfh\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9227e00f-e0cf-4a30-b287-1c645cf59134-sys\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9227e00f-e0cf-4a30-b287-1c645cf59134-sys\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92z5s\" (UniqueName: \"kubernetes.io/projected/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-api-access-92z5s\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9227e00f-e0cf-4a30-b287-1c645cf59134-root\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-textfile\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-accelerators-collector-config\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jh6\" (UniqueName: \"kubernetes.io/projected/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-kube-api-access-j6jh6\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.023671 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.024123 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.024123 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.024123 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9227e00f-e0cf-4a30-b287-1c645cf59134-root\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.024123 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023842    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9227e00f-e0cf-4a30-b287-1c645cf59134-metrics-client-ca\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.024123 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.024123 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.023981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-accelerators-collector-config\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.024366 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:48.024255    2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 23 18:17:48.024366 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:48.024355    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-openshift-state-metrics-tls podName:005317cb-c24c-4cd2-ab7c-d3fc844b3e36 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:48.524338283 +0000 UTC m=+64.372028793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-99qtp" (UID: "005317cb-c24c-4cd2-ab7c-d3fc844b3e36") : secret "openshift-state-metrics-tls" not found
Jul 23 18:17:48.025745 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.025717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-tls\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.025866 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.025801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9227e00f-e0cf-4a30-b287-1c645cf59134-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.025866 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.025857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.035566 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.035547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jh6\" (UniqueName: \"kubernetes.io/projected/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-kube-api-access-j6jh6\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.035715 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.035699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtfh\" (UniqueName: \"kubernetes.io/projected/9227e00f-e0cf-4a30-b287-1c645cf59134-kube-api-access-mhtfh\") pod \"node-exporter-z5q82\" (UID: \"9227e00f-e0cf-4a30-b287-1c645cf59134\") " pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.124998 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.124913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.124998 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.124981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.125220 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.125001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92z5s\" (UniqueName: \"kubernetes.io/projected/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-api-access-92z5s\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.125220 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.125033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.125220 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.125057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.125220 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.125076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.125448 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:48.125241    2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 23 18:17:48.125448 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:48.125322    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-tls podName:943a1fe3-2eb3-4b63-8de0-edadaf8c5aed nodeName:}" failed. No retries permitted until 2026-07-23 18:17:48.625308009 +0000 UTC m=+64.472998514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-vlgm7" (UID: "943a1fe3-2eb3-4b63-8de0-edadaf8c5aed") : secret "kube-state-metrics-tls" not found
Jul 23 18:17:48.125645 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.125612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.125833 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.125813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.125938 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.125919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.127389 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.127369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.133569 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.133549    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92z5s\" (UniqueName: \"kubernetes.io/projected/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-api-access-92z5s\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.183723 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.183696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z5q82"
Jul 23 18:17:48.193700 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:48.193676    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9227e00f_e0cf_4a30_b287_1c645cf59134.slice/crio-24dc59779fe33c13ecbe457b05bfc468d9efd01331b5077d992f0a6963ffa079 WatchSource:0}: Error finding container 24dc59779fe33c13ecbe457b05bfc468d9efd01331b5077d992f0a6963ffa079: Status 404 returned error can't find the container with id 24dc59779fe33c13ecbe457b05bfc468d9efd01331b5077d992f0a6963ffa079
Jul 23 18:17:48.527895 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.527862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.530234 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.530208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/005317cb-c24c-4cd2-ab7c-d3fc844b3e36-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-99qtp\" (UID: \"005317cb-c24c-4cd2-ab7c-d3fc844b3e36\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.628808 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.628771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.631764 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.631739    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/943a1fe3-2eb3-4b63-8de0-edadaf8c5aed-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vlgm7\" (UID: \"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.778459 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.778382    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"
Jul 23 18:17:48.810848 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.810819    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"
Jul 23 18:17:48.922617 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.922585    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp"]
Jul 23 18:17:48.925386 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.925361    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 18:17:48.928388 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.928365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:48.931125 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.931107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 23 18:17:48.931475 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.931451    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 23 18:17:48.931584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.931451    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gxctx\""
Jul 23 18:17:48.931650 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.931461    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 23 18:17:48.931715 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.931698    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 23 18:17:48.931809 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.931793    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 23 18:17:48.931911 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.931895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 23 18:17:48.931999 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.931983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 23 18:17:48.932200 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.932179    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 23 18:17:48.932325 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.932181    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 23 18:17:48.944445 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:48.944409    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 18:17:49.023478 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:49.023447    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005317cb_c24c_4cd2_ab7c_d3fc844b3e36.slice/crio-7d0d6b59bf7f98a8a04709f5679bb6dddfe221498d137933ffbad93a3b6d5791 WatchSource:0}: Error finding container 7d0d6b59bf7f98a8a04709f5679bb6dddfe221498d137933ffbad93a3b6d5791: Status 404 returned error can't find the container with id 7d0d6b59bf7f98a8a04709f5679bb6dddfe221498d137933ffbad93a3b6d5791
Jul 23 18:17:49.032560 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032533    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32de064e-90de-4385-82af-f74dbe5e8599-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.032688 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.032766 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.032819 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32de064e-90de-4385-82af-f74dbe5e8599-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.032819 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-config-volume\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.032914 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.032914 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32de064e-90de-4385-82af-f74dbe5e8599-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.033060 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clf4p\" (UniqueName: \"kubernetes.io/projected/32de064e-90de-4385-82af-f74dbe5e8599-kube-api-access-clf4p\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.033060 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.033060 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.032993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-web-config\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.033060 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.033020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32de064e-90de-4385-82af-f74dbe5e8599-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.033244 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.033058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.033244 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.033109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32de064e-90de-4385-82af-f74dbe5e8599-config-out\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.133921 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.133886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32de064e-90de-4385-82af-f74dbe5e8599-config-out\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.134039 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.133943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32de064e-90de-4385-82af-f74dbe5e8599-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.134039 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.134010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.134707 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.134041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.134707 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.134081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32de064e-90de-4385-82af-f74dbe5e8599-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.134707 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.134108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-config-volume\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.134707 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.134135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.134707 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:49.134262    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 23 18:17:49.134707 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:49.134358    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-main-tls podName:32de064e-90de-4385-82af-f74dbe5e8599 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:49.634336249 +0000 UTC m=+65.482026795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "32de064e-90de-4385-82af-f74dbe5e8599") : secret "alertmanager-main-tls" not found
Jul 23 18:17:49.135252 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.135216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32de064e-90de-4385-82af-f74dbe5e8599-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.135361 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.135274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clf4p\" (UniqueName: \"kubernetes.io/projected/32de064e-90de-4385-82af-f74dbe5e8599-kube-api-access-clf4p\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.135361 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.135328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.135361 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.135355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-web-config\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.135569 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.135386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32de064e-90de-4385-82af-f74dbe5e8599-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.135569 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.135416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.139148 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.136530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32de064e-90de-4385-82af-f74dbe5e8599-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.139148 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:17:49.136663    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32de064e-90de-4385-82af-f74dbe5e8599-alertmanager-trusted-ca-bundle podName:32de064e-90de-4385-82af-f74dbe5e8599 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:49.636643923 +0000 UTC m=+65.484334442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/32de064e-90de-4385-82af-f74dbe5e8599-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "32de064e-90de-4385-82af-f74dbe5e8599") : configmap references non-existent config key: ca-bundle.crt
Jul 23 18:17:49.139148 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.138104    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32de064e-90de-4385-82af-f74dbe5e8599-config-out\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.139148 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.138230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32de064e-90de-4385-82af-f74dbe5e8599-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.139148 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.138361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32de064e-90de-4385-82af-f74dbe5e8599-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.139467 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.139196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.139883 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.139840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.141260 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.141219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-web-config\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.141912 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.141513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.142366 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.142342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-config-volume\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.142461 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.142365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.145248 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.145227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clf4p\" (UniqueName: \"kubernetes.io/projected/32de064e-90de-4385-82af-f74dbe5e8599-kube-api-access-clf4p\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.176353 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.176240    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vlgm7"]
Jul 23 18:17:49.179681 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:49.179651    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943a1fe3_2eb3_4b63_8de0_edadaf8c5aed.slice/crio-c3897129407d9b219bca6c42bf3f692a0c189ab8175b5e589f62365aa9811dfd WatchSource:0}: Error finding container c3897129407d9b219bca6c42bf3f692a0c189ab8175b5e589f62365aa9811dfd: Status 404 returned error can't find the container with id c3897129407d9b219bca6c42bf3f692a0c189ab8175b5e589f62365aa9811dfd
Jul 23 18:17:49.185300 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.185254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7" event={"ID":"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed","Type":"ContainerStarted","Data":"c3897129407d9b219bca6c42bf3f692a0c189ab8175b5e589f62365aa9811dfd"}
Jul 23 18:17:49.186865 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.186838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z5q82" event={"ID":"9227e00f-e0cf-4a30-b287-1c645cf59134","Type":"ContainerStarted","Data":"66229b9b4bdd1af00d18ae0e19f16a25821380ca92f7000d20cc54ffaeff65d3"}
Jul 23 18:17:49.186967 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.186899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z5q82" event={"ID":"9227e00f-e0cf-4a30-b287-1c645cf59134","Type":"ContainerStarted","Data":"24dc59779fe33c13ecbe457b05bfc468d9efd01331b5077d992f0a6963ffa079"}
Jul 23 18:17:49.188869 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.188847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp" event={"ID":"005317cb-c24c-4cd2-ab7c-d3fc844b3e36","Type":"ContainerStarted","Data":"e0416913a4f9dbf938a94951bbed028f174a692762b20ba9d0f993226e749ebb"}
Jul 23 18:17:49.188958 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.188897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp" event={"ID":"005317cb-c24c-4cd2-ab7c-d3fc844b3e36","Type":"ContainerStarted","Data":"7d0d6b59bf7f98a8a04709f5679bb6dddfe221498d137933ffbad93a3b6d5791"}
Jul 23 18:17:49.641966 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.641935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32de064e-90de-4385-82af-f74dbe5e8599-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.642129 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.642022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.642764 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.642744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32de064e-90de-4385-82af-f74dbe5e8599-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.644349 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.644330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32de064e-90de-4385-82af-f74dbe5e8599-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32de064e-90de-4385-82af-f74dbe5e8599\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:49.841969 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:49.841886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 18:17:50.003076 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.003041    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 18:17:50.007808 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:50.007779    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32de064e_90de_4385_82af_f74dbe5e8599.slice/crio-ca54682ccfdd2e5caada20c8360522d034427e228009442777b9e05cb370c6b4 WatchSource:0}: Error finding container ca54682ccfdd2e5caada20c8360522d034427e228009442777b9e05cb370c6b4: Status 404 returned error can't find the container with id ca54682ccfdd2e5caada20c8360522d034427e228009442777b9e05cb370c6b4
Jul 23 18:17:50.194691 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.194602    2561 generic.go:356] "Generic (PLEG): container finished" podID="9227e00f-e0cf-4a30-b287-1c645cf59134" containerID="66229b9b4bdd1af00d18ae0e19f16a25821380ca92f7000d20cc54ffaeff65d3" exitCode=0
Jul 23 18:17:50.194850 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.194679    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z5q82" event={"ID":"9227e00f-e0cf-4a30-b287-1c645cf59134","Type":"ContainerDied","Data":"66229b9b4bdd1af00d18ae0e19f16a25821380ca92f7000d20cc54ffaeff65d3"}
Jul 23 18:17:50.196955 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.196929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp" event={"ID":"005317cb-c24c-4cd2-ab7c-d3fc844b3e36","Type":"ContainerStarted","Data":"fe8b16d662e9fcdfeff15f1a75f667907dc6dd30f2592ca633cee5f8bef16c1f"}
Jul 23 18:17:50.198184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.198161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32de064e-90de-4385-82af-f74dbe5e8599","Type":"ContainerStarted","Data":"ca54682ccfdd2e5caada20c8360522d034427e228009442777b9e05cb370c6b4"}
Jul 23 18:17:50.347644 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.347602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:50.350487 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.350465    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 18:17:50.360723 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.360694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ee12eab-d883-42ae-94cc-6618823fad3c-original-pull-secret\") pod \"global-pull-secret-syncer-vqch6\" (UID: \"4ee12eab-d883-42ae-94cc-6618823fad3c\") " pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:50.377719 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.377699    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqch6"
Jul 23 18:17:50.448743 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.448704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:50.451833 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.451809    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 18:17:50.462013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.461990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9413cd-4d10-4609-a427-55fcc8474d38-metrics-certs\") pod \"network-metrics-daemon-75jnc\" (UID: \"4e9413cd-4d10-4609-a427-55fcc8474d38\") " pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:50.549613 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.549575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:50.551950 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.551922    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 18:17:50.562837 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.562818    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 18:17:50.574798 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.574769    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kn7\" (UniqueName: \"kubernetes.io/projected/a4abfb91-e90d-450f-8a07-b6fa81d7a6d0-kube-api-access-n7kn7\") pod \"network-check-target-kwpb4\" (UID: \"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0\") " pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:50.579141 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.579122    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsfk8\""
Jul 23 18:17:50.587296 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.587266    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-75jnc"
Jul 23 18:17:50.838038 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.838016    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vqch6"]
Jul 23 18:17:50.850070 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:50.850026    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee12eab_d883_42ae_94cc_6618823fad3c.slice/crio-746254508e111fe9395f59566d138f3f1964a6e5364bd97802a487ac2e8f7f31 WatchSource:0}: Error finding container 746254508e111fe9395f59566d138f3f1964a6e5364bd97802a487ac2e8f7f31: Status 404 returned error can't find the container with id 746254508e111fe9395f59566d138f3f1964a6e5364bd97802a487ac2e8f7f31
Jul 23 18:17:50.860353 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.860331    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-75jnc"]
Jul 23 18:17:50.865110 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.865084    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"]
Jul 23 18:17:50.870218 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.869615    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:50.872920 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.872547    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 23 18:17:50.874532 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.874505    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p92cg\""
Jul 23 18:17:50.874832 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.874646    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w7wd2\""
Jul 23 18:17:50.874832 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.874598    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 23 18:17:50.874832 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.874808    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 23 18:17:50.875228 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.875202    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-77t8b0h69o82e\""
Jul 23 18:17:50.875504 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.875223    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 23 18:17:50.875706 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.875344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 23 18:17:50.878030 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:50.877762    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9413cd_4d10_4609_a427_55fcc8474d38.slice/crio-07701e5f6617dcb58e9b7fee68cd3a0a848ed4a369d94ef81e2e541953fa080e WatchSource:0}: Error finding container 07701e5f6617dcb58e9b7fee68cd3a0a848ed4a369d94ef81e2e541953fa080e: Status 404 returned error can't find the container with id 07701e5f6617dcb58e9b7fee68cd3a0a848ed4a369d94ef81e2e541953fa080e
Jul 23 18:17:50.881820 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.881587    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:50.888486 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.887962    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"]
Jul 23 18:17:50.952144 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.952048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:50.952272 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.952215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-tls\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:50.952272 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.952253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b89e84-d522-434f-b3c9-10131c183a9a-metrics-client-ca\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:50.952387 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.952313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:50.952387 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.952348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:50.952477 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.952386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmznj\" (UniqueName: \"kubernetes.io/projected/e6b89e84-d522-434f-b3c9-10131c183a9a-kube-api-access-wmznj\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:50.952477 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.952453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:50.952552 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:50.952499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-grpc-tls\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.040670 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.040632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kwpb4"]
Jul 23 18:17:51.046279 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:51.046245    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4abfb91_e90d_450f_8a07_b6fa81d7a6d0.slice/crio-9de1b354062221a58f10d283ec48abb895c0734b75683d948487e85827028452 WatchSource:0}: Error finding container 9de1b354062221a58f10d283ec48abb895c0734b75683d948487e85827028452: Status 404 returned error can't find the container with id 9de1b354062221a58f10d283ec48abb895c0734b75683d948487e85827028452
Jul 23 18:17:51.053088 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.053057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmznj\" (UniqueName: \"kubernetes.io/projected/e6b89e84-d522-434f-b3c9-10131c183a9a-kube-api-access-wmznj\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.053173 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.053112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.053173 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.053155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-grpc-tls\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.053294 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.053193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.053858 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.053415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-tls\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.053858 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.053453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b89e84-d522-434f-b3c9-10131c183a9a-metrics-client-ca\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.053858 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.053490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.054317 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.054277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b89e84-d522-434f-b3c9-10131c183a9a-metrics-client-ca\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.054628 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.054601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.056320 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.056263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.056877 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.056856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-grpc-tls\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.057337 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.057314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.057663 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.057631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.057875 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.057854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.058702 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.058682    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e6b89e84-d522-434f-b3c9-10131c183a9a-secret-thanos-querier-tls\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.062459 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.062432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmznj\" (UniqueName: \"kubernetes.io/projected/e6b89e84-d522-434f-b3c9-10131c183a9a-kube-api-access-wmznj\") pod \"thanos-querier-6cd98d6c6b-fpjv4\" (UID: \"e6b89e84-d522-434f-b3c9-10131c183a9a\") " pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.195731 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.195698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:17:51.204589 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.204490    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vqch6" event={"ID":"4ee12eab-d883-42ae-94cc-6618823fad3c","Type":"ContainerStarted","Data":"746254508e111fe9395f59566d138f3f1964a6e5364bd97802a487ac2e8f7f31"}
Jul 23 18:17:51.207881 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.207852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z5q82" event={"ID":"9227e00f-e0cf-4a30-b287-1c645cf59134","Type":"ContainerStarted","Data":"f118deed96b95b3b22cfaafe3f88a44ef67d518091ee9ce610c09f33b67b8f85"}
Jul 23 18:17:51.207996 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.207918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z5q82" event={"ID":"9227e00f-e0cf-4a30-b287-1c645cf59134","Type":"ContainerStarted","Data":"a5befcb8c72c738b7803516b20f4b45a55787c302ac926a2428e053936599262"}
Jul 23 18:17:51.211691 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.211658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp" event={"ID":"005317cb-c24c-4cd2-ab7c-d3fc844b3e36","Type":"ContainerStarted","Data":"6cafcc1e5301c3b44f1eeb74aef3dc9011265bcba88f3aeed336002adb4d0192"}
Jul 23 18:17:51.214515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.214489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7" event={"ID":"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed","Type":"ContainerStarted","Data":"1e9cc8b4147efce19c6770a89f277e8ddf7b76870d6f0377f1a03575bad5b44a"}
Jul 23 18:17:51.214635 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.214546    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7" event={"ID":"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed","Type":"ContainerStarted","Data":"dce4c1ffa42b5fc606738a207c48e340653f11ad82aa7b8b329586c66a1ab05b"}
Jul 23 18:17:51.214635 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.214573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7" event={"ID":"943a1fe3-2eb3-4b63-8de0-edadaf8c5aed","Type":"ContainerStarted","Data":"aab7fe82bda46cd56b65c7e525e852acf69ade1317c08674594f13c3c11130d3"}
Jul 23 18:17:51.215986 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.215964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kwpb4" event={"ID":"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0","Type":"ContainerStarted","Data":"9de1b354062221a58f10d283ec48abb895c0734b75683d948487e85827028452"}
Jul 23 18:17:51.217467 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.217423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-75jnc" event={"ID":"4e9413cd-4d10-4609-a427-55fcc8474d38","Type":"ContainerStarted","Data":"07701e5f6617dcb58e9b7fee68cd3a0a848ed4a369d94ef81e2e541953fa080e"}
Jul 23 18:17:51.228354 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.228300    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z5q82" podStartSLOduration=3.293352465 podStartE2EDuration="4.228269417s" totalImagesPullingTime="859.15789ms" totalInitContainerRuntime="75.759062ms" isStatefulPod=true podCreationTimestamp="2026-07-23 18:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:51.225990966 +0000 UTC m=+67.073681534" watchObservedRunningTime="2026-07-23 18:17:51.228269417 +0000 UTC m=+67.075959924"
Jul 23 18:17:51.245101 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.244821    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-99qtp" podStartSLOduration=2.7698888139999998 podStartE2EDuration="4.244800969s" totalImagesPullingTime="1.474912155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:51.244462656 +0000 UTC m=+67.092153185" watchObservedRunningTime="2026-07-23 18:17:51.244800969 +0000 UTC m=+67.092491508"
Jul 23 18:17:51.266905 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.266843    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-vlgm7" podStartSLOduration=2.781263229 podStartE2EDuration="4.266822532s" totalImagesPullingTime="1.485559303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:51.266257442 +0000 UTC m=+67.113947981" watchObservedRunningTime="2026-07-23 18:17:51.266822532 +0000 UTC m=+67.114513064"
Jul 23 18:17:51.353742 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:51.353700    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"]
Jul 23 18:17:51.619169 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:51.619101    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b89e84_d522_434f_b3c9_10131c183a9a.slice/crio-f582c36b030dc96a9ab9d28b735347ed1ca21530c6a82ac94e362e26afe3f722 WatchSource:0}: Error finding container f582c36b030dc96a9ab9d28b735347ed1ca21530c6a82ac94e362e26afe3f722: Status 404 returned error can't find the container with id f582c36b030dc96a9ab9d28b735347ed1ca21530c6a82ac94e362e26afe3f722
Jul 23 18:17:52.155435 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.155398    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c58cdbd8-f88kr"]
Jul 23 18:17:52.158013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.157992    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.161277 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.161249    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 23 18:17:52.162439 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.162418    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 18:17:52.162526 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.162442    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 23 18:17:52.162774 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.162754    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7txdg\""
Jul 23 18:17:52.163071 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.163041    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a8tts71dl2itt\""
Jul 23 18:17:52.163251 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.163234    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 23 18:17:52.168170 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.168096    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c58cdbd8-f88kr"]
Jul 23 18:17:52.225945 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.225904    2561 generic.go:356] "Generic (PLEG): container finished" podID="32de064e-90de-4385-82af-f74dbe5e8599" containerID="23f5c461e3d2843fbce4bb0693442b9bebdff5cc259c9aad73c23dee83ef79e2" exitCode=0
Jul 23 18:17:52.226083 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.226045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32de064e-90de-4385-82af-f74dbe5e8599","Type":"ContainerDied","Data":"23f5c461e3d2843fbce4bb0693442b9bebdff5cc259c9aad73c23dee83ef79e2"}
Jul 23 18:17:52.230140 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.228938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4" event={"ID":"e6b89e84-d522-434f-b3c9-10131c183a9a","Type":"ContainerStarted","Data":"f582c36b030dc96a9ab9d28b735347ed1ca21530c6a82ac94e362e26afe3f722"}
Jul 23 18:17:52.268184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.268135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k57gp\" (UniqueName: \"kubernetes.io/projected/efde315e-f94f-4e8c-8335-9fd9f09074c6-kube-api-access-k57gp\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.268184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.268218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efde315e-f94f-4e8c-8335-9fd9f09074c6-client-ca-bundle\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.268184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.268263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efde315e-f94f-4e8c-8335-9fd9f09074c6-metrics-server-audit-profiles\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.268184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.268332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efde315e-f94f-4e8c-8335-9fd9f09074c6-secret-metrics-server-client-certs\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.268184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.268356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde315e-f94f-4e8c-8335-9fd9f09074c6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.268184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.268382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efde315e-f94f-4e8c-8335-9fd9f09074c6-secret-metrics-server-tls\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.268184 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.268422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efde315e-f94f-4e8c-8335-9fd9f09074c6-audit-log\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.369592 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.369544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efde315e-f94f-4e8c-8335-9fd9f09074c6-client-ca-bundle\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.369785 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.369699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efde315e-f94f-4e8c-8335-9fd9f09074c6-metrics-server-audit-profiles\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.369785 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.369774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efde315e-f94f-4e8c-8335-9fd9f09074c6-secret-metrics-server-client-certs\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.369905 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.369801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde315e-f94f-4e8c-8335-9fd9f09074c6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.369905 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.369830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efde315e-f94f-4e8c-8335-9fd9f09074c6-secret-metrics-server-tls\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.369994 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.369897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efde315e-f94f-4e8c-8335-9fd9f09074c6-audit-log\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.370046 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.370014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k57gp\" (UniqueName: \"kubernetes.io/projected/efde315e-f94f-4e8c-8335-9fd9f09074c6-kube-api-access-k57gp\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.371302 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.371254    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efde315e-f94f-4e8c-8335-9fd9f09074c6-audit-log\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.371575 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.371551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde315e-f94f-4e8c-8335-9fd9f09074c6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.372161 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.372111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efde315e-f94f-4e8c-8335-9fd9f09074c6-metrics-server-audit-profiles\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.374277 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.374222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efde315e-f94f-4e8c-8335-9fd9f09074c6-secret-metrics-server-client-certs\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.374587 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.374566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efde315e-f94f-4e8c-8335-9fd9f09074c6-client-ca-bundle\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.374753 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.374732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efde315e-f94f-4e8c-8335-9fd9f09074c6-secret-metrics-server-tls\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.379643 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.379599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k57gp\" (UniqueName: \"kubernetes.io/projected/efde315e-f94f-4e8c-8335-9fd9f09074c6-kube-api-access-k57gp\") pod \"metrics-server-5c58cdbd8-f88kr\" (UID: \"efde315e-f94f-4e8c-8335-9fd9f09074c6\") " pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.471853 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.471815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:17:52.652688 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.652655    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c58cdbd8-f88kr"]
Jul 23 18:17:52.657601 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:52.657560    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefde315e_f94f_4e8c_8335_9fd9f09074c6.slice/crio-bd5a96c0e20ed1383b785230d7cc0e7bf821d2afac89f1a41afbdadd1abd6dcb WatchSource:0}: Error finding container bd5a96c0e20ed1383b785230d7cc0e7bf821d2afac89f1a41afbdadd1abd6dcb: Status 404 returned error can't find the container with id bd5a96c0e20ed1383b785230d7cc0e7bf821d2afac89f1a41afbdadd1abd6dcb
Jul 23 18:17:52.679370 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.678335    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"]
Jul 23 18:17:52.681517 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.681140    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:52.682506 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.681806    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bd9f8558c-jc5g8"]
Jul 23 18:17:52.685645 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.684051    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.685645 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.685253    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r2lc9\""
Jul 23 18:17:52.685645 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.685515    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 23 18:17:52.692374 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.692315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"]
Jul 23 18:17:52.698267 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.698201    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bd9f8558c-jc5g8"]
Jul 23 18:17:52.774804 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.774520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpvw\" (UniqueName: \"kubernetes.io/projected/aeda98e7-9719-4a8d-9281-ef459dee0ca0-kube-api-access-jlpvw\") pod \"monitoring-plugin-7b7b7bc8bf-cfhkw\" (UID: \"aeda98e7-9719-4a8d-9281-ef459dee0ca0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:52.774804 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.774681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aeda98e7-9719-4a8d-9281-ef459dee0ca0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cfhkw\" (UID: \"aeda98e7-9719-4a8d-9281-ef459dee0ca0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:52.774804 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.774715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-serving-cert\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.774804 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.774751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-oauth-config\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.775122 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.774804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-service-ca\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.775122 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.774841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7w68\" (UniqueName: \"kubernetes.io/projected/8d7074c5-f4de-4bc8-8681-a3913f3156d5-kube-api-access-h7w68\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.775122 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.774870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-oauth-serving-cert\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.775122 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.774894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-config\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.775122 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.774919    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-trusted-ca-bundle\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.875959 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.875920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-service-ca\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.876142 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.875984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w68\" (UniqueName: \"kubernetes.io/projected/8d7074c5-f4de-4bc8-8681-a3913f3156d5-kube-api-access-h7w68\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.876142 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.876016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-oauth-serving-cert\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.876142 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.876043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-config\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.876142 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.876070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-trusted-ca-bundle\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.876142 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.876115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpvw\" (UniqueName: \"kubernetes.io/projected/aeda98e7-9719-4a8d-9281-ef459dee0ca0-kube-api-access-jlpvw\") pod \"monitoring-plugin-7b7b7bc8bf-cfhkw\" (UID: \"aeda98e7-9719-4a8d-9281-ef459dee0ca0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:52.876448 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.876168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aeda98e7-9719-4a8d-9281-ef459dee0ca0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cfhkw\" (UID: \"aeda98e7-9719-4a8d-9281-ef459dee0ca0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:52.876448 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.876193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-serving-cert\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.876448 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.876230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-oauth-config\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.877406 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.877353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-config\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.877605 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.877580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-oauth-serving-cert\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.878026 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.878001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-trusted-ca-bundle\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.878144 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.878122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-service-ca\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.879344 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.879324    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-oauth-config\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.880236 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.880181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-serving-cert\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.880656 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.880612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aeda98e7-9719-4a8d-9281-ef459dee0ca0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cfhkw\" (UID: \"aeda98e7-9719-4a8d-9281-ef459dee0ca0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:52.887349 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.887306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7w68\" (UniqueName: \"kubernetes.io/projected/8d7074c5-f4de-4bc8-8681-a3913f3156d5-kube-api-access-h7w68\") pod \"console-bd9f8558c-jc5g8\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") " pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:52.889276 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.889250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpvw\" (UniqueName: \"kubernetes.io/projected/aeda98e7-9719-4a8d-9281-ef459dee0ca0-kube-api-access-jlpvw\") pod \"monitoring-plugin-7b7b7bc8bf-cfhkw\" (UID: \"aeda98e7-9719-4a8d-9281-ef459dee0ca0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:52.999953 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:52.999912    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:53.007820 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:53.007646    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:17:53.191528 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:53.191331    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bd9f8558c-jc5g8"]
Jul 23 18:17:53.218107 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:53.218080    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"]
Jul 23 18:17:53.235765 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:53.235692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr" event={"ID":"efde315e-f94f-4e8c-8335-9fd9f09074c6","Type":"ContainerStarted","Data":"bd5a96c0e20ed1383b785230d7cc0e7bf821d2afac89f1a41afbdadd1abd6dcb"}
Jul 23 18:17:53.465441 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:53.465401    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7074c5_f4de_4bc8_8681_a3913f3156d5.slice/crio-263ebd14472549128a7b0e69d9e6f07371171a369be2c0606b622486f315e353 WatchSource:0}: Error finding container 263ebd14472549128a7b0e69d9e6f07371171a369be2c0606b622486f315e353: Status 404 returned error can't find the container with id 263ebd14472549128a7b0e69d9e6f07371171a369be2c0606b622486f315e353
Jul 23 18:17:53.465983 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:53.465951    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeda98e7_9719_4a8d_9281_ef459dee0ca0.slice/crio-90c8c84b65a133773306a70b08baf2210c745c0a4c082e1b46be92105f90b972 WatchSource:0}: Error finding container 90c8c84b65a133773306a70b08baf2210c745c0a4c082e1b46be92105f90b972: Status 404 returned error can't find the container with id 90c8c84b65a133773306a70b08baf2210c745c0a4c082e1b46be92105f90b972
Jul 23 18:17:54.183936 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.183903    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 18:17:54.187033 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.187010    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.191454 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.191426    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 18:17:54.191800 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.191674    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5gvh84sij6mfr\""
Jul 23 18:17:54.191800 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.191726    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 18:17:54.192394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.191894    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 18:17:54.192730 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.192710    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 18:17:54.193012 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.192917    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 18:17:54.193123 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.193084    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 18:17:54.193197 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.193144    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 18:17:54.193262 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.193216    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x8djk\""
Jul 23 18:17:54.193783 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.193689    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 18:17:54.194824 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.193925    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 18:17:54.194824 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.194053    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 18:17:54.194824 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.194560    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 18:17:54.194824 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.194630    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 18:17:54.195075 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.194850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 18:17:54.202243 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.199592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 18:17:54.218668 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.218639    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 18:17:54.246713 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.246673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd9f8558c-jc5g8" event={"ID":"8d7074c5-f4de-4bc8-8681-a3913f3156d5","Type":"ContainerStarted","Data":"a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f"}
Jul 23 18:17:54.246953 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.246928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd9f8558c-jc5g8" event={"ID":"8d7074c5-f4de-4bc8-8681-a3913f3156d5","Type":"ContainerStarted","Data":"263ebd14472549128a7b0e69d9e6f07371171a369be2c0606b622486f315e353"}
Jul 23 18:17:54.248297 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.248250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw" event={"ID":"aeda98e7-9719-4a8d-9281-ef459dee0ca0","Type":"ContainerStarted","Data":"90c8c84b65a133773306a70b08baf2210c745c0a4c082e1b46be92105f90b972"}
Jul 23 18:17:54.250007 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.249979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-75jnc" event={"ID":"4e9413cd-4d10-4609-a427-55fcc8474d38","Type":"ContainerStarted","Data":"22772987c0ded16a86f228837578c1378d5df95a6a177b53cbb1cb77119b34d9"}
Jul 23 18:17:54.289503 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289515    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-config\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289636    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289957 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289957 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289957 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289957 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwsf\" (UniqueName: \"kubernetes.io/projected/b428ca74-e113-4736-b602-6a4433562e84-kube-api-access-4cwsf\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289957 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.289957 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.290258 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289958    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.290258 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.289988    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.290258 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.290022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.290258 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.290073    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-web-config\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.290258 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.290100    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b428ca74-e113-4736-b602-6a4433562e84-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.290258 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.290135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b428ca74-e113-4736-b602-6a4433562e84-config-out\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.290258 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.290178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.290258 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.290229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.391511 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.391412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.391511 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.391464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.391511 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.391496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.391511 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.391519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.391920 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.391892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-web-config\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.391994 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.391959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b428ca74-e113-4736-b602-6a4433562e84-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392050 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b428ca74-e113-4736-b602-6a4433562e84-config-out\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392050 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392167 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392167 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392167 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392303 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-config\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392303 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392303 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392303 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392427 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392427 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392360    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392427 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392368    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.392511 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.393019 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.392964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.393019 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.393011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwsf\" (UniqueName: \"kubernetes.io/projected/b428ca74-e113-4736-b602-6a4433562e84-kube-api-access-4cwsf\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.393186 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.393051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.393780 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.393751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.393896 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.393857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.398503 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.397442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.398503 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.397914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.398503 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.398003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.398503 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.398153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b428ca74-e113-4736-b602-6a4433562e84-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.399442 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.399360    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b428ca74-e113-4736-b602-6a4433562e84-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.400561 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.400043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b428ca74-e113-4736-b602-6a4433562e84-config-out\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.400561 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.400512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-web-config\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.401594 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.401395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.402783 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.401898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-config\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.402783 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.402042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.402783 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.402219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b428ca74-e113-4736-b602-6a4433562e84-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.402783 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.402340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.402783 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.402736    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.403103 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.402847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.404077 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.404052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b428ca74-e113-4736-b602-6a4433562e84-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.404172 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.404083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwsf\" (UniqueName: \"kubernetes.io/projected/b428ca74-e113-4736-b602-6a4433562e84-kube-api-access-4cwsf\") pod \"prometheus-k8s-0\" (UID: \"b428ca74-e113-4736-b602-6a4433562e84\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:54.504674 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:54.504635    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:17:57.780316 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:57.780240    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bd9f8558c-jc5g8" podStartSLOduration=5.780223484 podStartE2EDuration="5.780223484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:54.280022524 +0000 UTC m=+70.127713054" watchObservedRunningTime="2026-07-23 18:17:57.780223484 +0000 UTC m=+73.627914040"
Jul 23 18:17:57.780787 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:57.780573    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 18:17:58.011001 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:17:58.010968    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb428ca74_e113_4736_b602_6a4433562e84.slice/crio-8a09afbe917a559b8a36329f9662e31e202ff98b3ed622779675f6c3d0df5d93 WatchSource:0}: Error finding container 8a09afbe917a559b8a36329f9662e31e202ff98b3ed622779675f6c3d0df5d93: Status 404 returned error can't find the container with id 8a09afbe917a559b8a36329f9662e31e202ff98b3ed622779675f6c3d0df5d93
Jul 23 18:17:58.270467 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:58.270218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-75jnc" event={"ID":"4e9413cd-4d10-4609-a427-55fcc8474d38","Type":"ContainerStarted","Data":"ab470ae65ac6d1a6e57e704db599b1441be64a6922196c317521c4492c2fcf6f"}
Jul 23 18:17:58.272101 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:58.272067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b428ca74-e113-4736-b602-6a4433562e84","Type":"ContainerStarted","Data":"8a09afbe917a559b8a36329f9662e31e202ff98b3ed622779675f6c3d0df5d93"}
Jul 23 18:17:58.289985 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:58.289916    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-75jnc" podStartSLOduration=70.662740194 podStartE2EDuration="1m13.289892969s" totalImagesPullingTime="2.627152775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:58.288180598 +0000 UTC m=+74.135871126" watchObservedRunningTime="2026-07-23 18:17:58.289892969 +0000 UTC m=+74.137583498"
Jul 23 18:17:59.281698 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.281651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw" event={"ID":"aeda98e7-9719-4a8d-9281-ef459dee0ca0","Type":"ContainerStarted","Data":"212265ddf43cbb16b6e37efe3cc41deb1568975557f5be9f63ba0c97a0adc2c3"}
Jul 23 18:17:59.282136 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.281981    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:59.283767 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.283736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kwpb4" event={"ID":"a4abfb91-e90d-450f-8a07-b6fa81d7a6d0","Type":"ContainerStarted","Data":"c43b94c0f62e41216cc1a2f9d33222b7d8e36b246f26d59b9d3a07365622fc27"}
Jul 23 18:17:59.284409 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.284360    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:17:59.288963 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.288738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32de064e-90de-4385-82af-f74dbe5e8599","Type":"ContainerStarted","Data":"dbddd4a5a6026c94fced59db48e1f4be87143d60c928db967e5cd335f9f63127"}
Jul 23 18:17:59.288963 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.288807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32de064e-90de-4385-82af-f74dbe5e8599","Type":"ContainerStarted","Data":"0da38f846bb92ab0086b3f1a1eb81d17b79152965397fbc2b69776e82e85e347"}
Jul 23 18:17:59.288963 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.288846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32de064e-90de-4385-82af-f74dbe5e8599","Type":"ContainerStarted","Data":"5c3951e23096818dae53d62487eec059d0801d9e0e32836d9291ed6b87f7d059"}
Jul 23 18:17:59.288963 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.288882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32de064e-90de-4385-82af-f74dbe5e8599","Type":"ContainerStarted","Data":"85cba6b80f3af2987715173cba9f26e5a1959cc2453190b05c5bc5568c67a914"}
Jul 23 18:17:59.288963 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.288918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32de064e-90de-4385-82af-f74dbe5e8599","Type":"ContainerStarted","Data":"4fda17b4165ea493c4ce7319a6ce7ded125049b3d8f1aa75d96eda114fa4c184"}
Jul 23 18:17:59.290389 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.290347    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw"
Jul 23 18:17:59.290947 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.290923    2561 generic.go:356] "Generic (PLEG): container finished" podID="b428ca74-e113-4736-b602-6a4433562e84" containerID="36d17fdda7fdee8271a134b15f9f11ce533f2f139f099c2819a5e9bfb92132d2" exitCode=0
Jul 23 18:17:59.291054 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.291035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b428ca74-e113-4736-b602-6a4433562e84","Type":"ContainerDied","Data":"36d17fdda7fdee8271a134b15f9f11ce533f2f139f099c2819a5e9bfb92132d2"}
Jul 23 18:17:59.293703 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.293680    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4" event={"ID":"e6b89e84-d522-434f-b3c9-10131c183a9a","Type":"ContainerStarted","Data":"ab04e12beebb5cc38908132b46e732e73e420b5a3a1f164777b85ba33fe33ea9"}
Jul 23 18:17:59.293803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.293751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4" event={"ID":"e6b89e84-d522-434f-b3c9-10131c183a9a","Type":"ContainerStarted","Data":"28a6c8d4ef80e7e61103e8f3c1aec127938750a75e6a0ebc962ec5bd5223dd16"}
Jul 23 18:17:59.293803 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.293795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4" event={"ID":"e6b89e84-d522-434f-b3c9-10131c183a9a","Type":"ContainerStarted","Data":"33b4d0f411811e04e2289437672fcad6d3ef88d25729f57d5fd0dfcd2e254c03"}
Jul 23 18:17:59.295741 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.295712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vqch6" event={"ID":"4ee12eab-d883-42ae-94cc-6618823fad3c","Type":"ContainerStarted","Data":"0e2596baead3b43c954b801f7e0d8d6705fa7f5f7d71792f5be7ef59fbf6fad0"}
Jul 23 18:17:59.298216 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.298189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr" event={"ID":"efde315e-f94f-4e8c-8335-9fd9f09074c6","Type":"ContainerStarted","Data":"f42daf61685cd97539a39860b6729866f32de089e074b81d832d629321aedfa4"}
Jul 23 18:17:59.300804 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.300760    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cfhkw" podStartSLOduration=2.407104464 podStartE2EDuration="7.300747404s" totalImagesPullingTime="4.89364294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:59.298300062 +0000 UTC m=+75.145990586" watchObservedRunningTime="2026-07-23 18:17:59.300747404 +0000 UTC m=+75.148437930"
Jul 23 18:17:59.349052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.348999    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vqch6" podStartSLOduration=66.851464843 podStartE2EDuration="1m14.348978035s" totalImagesPullingTime="7.497513192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:59.315448467 +0000 UTC m=+75.163138995" watchObservedRunningTime="2026-07-23 18:17:59.348978035 +0000 UTC m=+75.196668563"
Jul 23 18:17:59.364737 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.364686    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kwpb4" podStartSLOduration=67.794956773 podStartE2EDuration="1m14.364671475s" totalImagesPullingTime="6.569714702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:59.364608511 +0000 UTC m=+75.212299039" watchObservedRunningTime="2026-07-23 18:17:59.364671475 +0000 UTC m=+75.212362001"
Jul 23 18:17:59.408942 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:17:59.408888    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr" podStartSLOduration=1.7122733060000002 podStartE2EDuration="7.408871389s" totalImagesPullingTime="5.696598083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:59.407579822 +0000 UTC m=+75.255270349" watchObservedRunningTime="2026-07-23 18:17:59.408871389 +0000 UTC m=+75.256561916"
Jul 23 18:18:00.306743 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:00.306703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32de064e-90de-4385-82af-f74dbe5e8599","Type":"ContainerStarted","Data":"b348e0411319fccae096ea55fc90927178a207ed9b8e6817dcf41bbd1e092370"}
Jul 23 18:18:00.310106 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:00.309964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4" event={"ID":"e6b89e84-d522-434f-b3c9-10131c183a9a","Type":"ContainerStarted","Data":"0b392a3cc237eff46b0d63fc8461b7ea178572a7e8d357797ab440c68357b91b"}
Jul 23 18:18:00.310106 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:00.310040    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4" event={"ID":"e6b89e84-d522-434f-b3c9-10131c183a9a","Type":"ContainerStarted","Data":"f40caf8256f54a5f867cbe0ac81a8b5109cf718753cb9a6b30b6c76fea6f9cda"}
Jul 23 18:18:00.310106 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:00.310081    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4" event={"ID":"e6b89e84-d522-434f-b3c9-10131c183a9a","Type":"ContainerStarted","Data":"0932a604e2aa9bed84a96761480c7d77d461deb73a10e7a184bc91f71945e80c"}
Jul 23 18:18:00.337875 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:00.337824    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.747182678 podStartE2EDuration="12.337807544s" totalImagesPullingTime="8.566594559s" totalInitContainerRuntime="24.030307ms" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:18:00.336355908 +0000 UTC m=+76.184046436" watchObservedRunningTime="2026-07-23 18:18:00.337807544 +0000 UTC m=+76.185498069"
Jul 23 18:18:00.363428 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:00.363339    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4" podStartSLOduration=2.671637223 podStartE2EDuration="10.363317477s" totalImagesPullingTime="7.691680254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:18:00.363220635 +0000 UTC m=+76.210911167" watchObservedRunningTime="2026-07-23 18:18:00.363317477 +0000 UTC m=+76.211008005"
Jul 23 18:18:01.315223 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:01.315132    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:18:03.008723 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:03.008685    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:18:03.009086 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:03.008753    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:18:03.013446 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:03.013426    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:18:03.327960 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:03.327900    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:18:03.382202 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:03.382170    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b85b9bbb7-f2vqq"]
Jul 23 18:18:04.336050 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:04.336015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b428ca74-e113-4736-b602-6a4433562e84","Type":"ContainerStarted","Data":"98b76d957224e7a05f5b4633896efc88f065ea06e74747f39c1e8585b8a0437e"}
Jul 23 18:18:04.336485 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:04.336118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b428ca74-e113-4736-b602-6a4433562e84","Type":"ContainerStarted","Data":"4386d78acf9d9fd6e97a52c44d48013ed7f9edfa2fbbbee041a297c4e069e525"}
Jul 23 18:18:04.336485 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:04.336149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b428ca74-e113-4736-b602-6a4433562e84","Type":"ContainerStarted","Data":"c63b134e1b3ba882eca9b92170df1908348b4a2a72e48d0ce699af461e4da9df"}
Jul 23 18:18:04.336485 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:04.336173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b428ca74-e113-4736-b602-6a4433562e84","Type":"ContainerStarted","Data":"22efaff967d7de24f6acfc4a38d126f8a7376ff3403f6a50af23735533c40cb9"}
Jul 23 18:18:04.336485 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:04.336194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b428ca74-e113-4736-b602-6a4433562e84","Type":"ContainerStarted","Data":"6e44639f1b9662177eed365477085fe3b65ca55cf9b74c319301f8e9b1343a35"}
Jul 23 18:18:04.336485 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:04.336217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b428ca74-e113-4736-b602-6a4433562e84","Type":"ContainerStarted","Data":"8c4831823435a2df28748f970f27e7fe13da9936d57545594ef8b223fc3cf92a"}
Jul 23 18:18:04.367867 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:04.367815    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.390302232 podStartE2EDuration="10.367799917s" totalImagesPullingTime="3.930899158s" totalInitContainerRuntime="46.598527ms" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:18:04.364937958 +0000 UTC m=+80.212628485" watchObservedRunningTime="2026-07-23 18:18:04.367799917 +0000 UTC m=+80.215490443"
Jul 23 18:18:04.505174 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:04.505138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:18:07.325630 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:07.325517    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6cd98d6c6b-fpjv4"
Jul 23 18:18:12.472338 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:12.472298    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:18:12.472812 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:12.472360    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:18:28.421368 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:28.421305    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b85b9bbb7-f2vqq" podUID="d3bd374a-43b1-45f7-bb42-39c0652ed26e" containerName="console" containerID="cri-o://e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870" gracePeriod=15
Jul 23 18:18:29.188628 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.188602    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:18:29.309846 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.309767    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-trusted-ca-bundle\") pod \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") "
Jul 23 18:18:29.309846 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.309818    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-oauth-serving-cert\") pod \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") "
Jul 23 18:18:29.310052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.309849    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlxt6\" (UniqueName: \"kubernetes.io/projected/d3bd374a-43b1-45f7-bb42-39c0652ed26e-kube-api-access-wlxt6\") pod \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") "
Jul 23 18:18:29.310052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.309893    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-oauth-config\") pod \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") "
Jul 23 18:18:29.310052 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.309923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-config\") pod \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") "
Jul 23 18:18:29.310202 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.310066    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-service-ca\") pod \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") "
Jul 23 18:18:29.310202 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.310131    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-serving-cert\") pod \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\" (UID: \"d3bd374a-43b1-45f7-bb42-39c0652ed26e\") "
Jul 23 18:18:29.310330 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.310182    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d3bd374a-43b1-45f7-bb42-39c0652ed26e" (UID: "d3bd374a-43b1-45f7-bb42-39c0652ed26e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:18:29.310398 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.310322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d3bd374a-43b1-45f7-bb42-39c0652ed26e" (UID: "d3bd374a-43b1-45f7-bb42-39c0652ed26e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:18:29.310469 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.310441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-service-ca" (OuterVolumeSpecName: "service-ca") pod "d3bd374a-43b1-45f7-bb42-39c0652ed26e" (UID: "d3bd374a-43b1-45f7-bb42-39c0652ed26e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:18:29.310600 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.310530    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-trusted-ca-bundle\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:18:29.310600 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.310548    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-oauth-serving-cert\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:18:29.310600 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.310563    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:18:29.310600 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.310591    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-config" (OuterVolumeSpecName: "console-config") pod "d3bd374a-43b1-45f7-bb42-39c0652ed26e" (UID: "d3bd374a-43b1-45f7-bb42-39c0652ed26e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:18:29.312397 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.312372    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d3bd374a-43b1-45f7-bb42-39c0652ed26e" (UID: "d3bd374a-43b1-45f7-bb42-39c0652ed26e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:18:29.312599 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.312580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bd374a-43b1-45f7-bb42-39c0652ed26e-kube-api-access-wlxt6" (OuterVolumeSpecName: "kube-api-access-wlxt6") pod "d3bd374a-43b1-45f7-bb42-39c0652ed26e" (UID: "d3bd374a-43b1-45f7-bb42-39c0652ed26e"). InnerVolumeSpecName "kube-api-access-wlxt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:18:29.312649 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.312584    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d3bd374a-43b1-45f7-bb42-39c0652ed26e" (UID: "d3bd374a-43b1-45f7-bb42-39c0652ed26e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:18:29.411639 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.411604    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-serving-cert\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:18:29.411639 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.411633    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlxt6\" (UniqueName: \"kubernetes.io/projected/d3bd374a-43b1-45f7-bb42-39c0652ed26e-kube-api-access-wlxt6\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:18:29.411639 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.411642    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-oauth-config\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:18:29.411849 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.411651    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3bd374a-43b1-45f7-bb42-39c0652ed26e-console-config\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:18:29.438340 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.438306    2561 generic.go:356] "Generic (PLEG): container finished" podID="d3bd374a-43b1-45f7-bb42-39c0652ed26e" containerID="e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870" exitCode=0
Jul 23 18:18:29.438757 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.438354    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b85b9bbb7-f2vqq"
Jul 23 18:18:29.438757 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.438395    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b85b9bbb7-f2vqq" event={"ID":"d3bd374a-43b1-45f7-bb42-39c0652ed26e","Type":"ContainerDied","Data":"e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870"}
Jul 23 18:18:29.438757 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.438459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b85b9bbb7-f2vqq" event={"ID":"d3bd374a-43b1-45f7-bb42-39c0652ed26e","Type":"ContainerDied","Data":"7cc3898e1b05fd9a815512a5e7b81643884fd825844f136536523651cb8ad6bc"}
Jul 23 18:18:29.438757 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.438491    2561 scope.go:117] "RemoveContainer" containerID="e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870"
Jul 23 18:18:29.458565 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.458542    2561 scope.go:117] "RemoveContainer" containerID="e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870"
Jul 23 18:18:29.458928 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:18:29.458901    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870\": container with ID starting with e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870 not found: ID does not exist" containerID="e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870"
Jul 23 18:18:29.459000 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.458942    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870"} err="failed to get container status \"e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870\": rpc error: code = NotFound desc = could not find container \"e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870\": container with ID starting with e181cd5371143e8e9b61c8f3e67ea594885383bf76672f183981aff3a4237870 not found: ID does not exist"
Jul 23 18:18:29.460164 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.460139    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b85b9bbb7-f2vqq"]
Jul 23 18:18:29.465118 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:29.465098    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b85b9bbb7-f2vqq"]
Jul 23 18:18:30.762047 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:30.762005    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bd374a-43b1-45f7-bb42-39c0652ed26e" path="/var/lib/kubelet/pods/d3bd374a-43b1-45f7-bb42-39c0652ed26e/volumes"
Jul 23 18:18:31.317869 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:31.317838    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kwpb4"
Jul 23 18:18:32.477323 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:32.477271    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:18:32.481109 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:32.481087    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c58cdbd8-f88kr"
Jul 23 18:18:54.506111 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:54.506070    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:18:54.525743 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:54.525719    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:18:54.563152 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:18:54.563125    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 18:19:12.186587 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.186551    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-77f855b89f-t6ps6"]
Jul 23 18:19:12.187267 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.187234    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bd374a-43b1-45f7-bb42-39c0652ed26e" containerName="console"
Jul 23 18:19:12.187267 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.187259    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bd374a-43b1-45f7-bb42-39c0652ed26e" containerName="console"
Jul 23 18:19:12.187440 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.187364    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bd374a-43b1-45f7-bb42-39c0652ed26e" containerName="console"
Jul 23 18:19:12.189651 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.189632    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.192764 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.192743    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 23 18:19:12.193521 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.193499    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 23 18:19:12.200515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.200493    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 23 18:19:12.201156 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.200766    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 23 18:19:12.201523 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.200871    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 23 18:19:12.201618 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.200922    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-njfwx\""
Jul 23 18:19:12.206795 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.206769    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 23 18:19:12.207456 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.207437    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77f855b89f-t6ps6"]
Jul 23 18:19:12.282594 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.282558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-federate-client-tls\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.282756 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.282598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784ab2ff-0096-4a1d-958d-734cdc1ead62-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.282756 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.282674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.282756 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.282720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784ab2ff-0096-4a1d-958d-734cdc1ead62-serving-certs-ca-bundle\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.282756 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.282746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjq8k\" (UniqueName: \"kubernetes.io/projected/784ab2ff-0096-4a1d-958d-734cdc1ead62-kube-api-access-fjq8k\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.282962 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.282868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-telemeter-client-tls\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.282962 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.282918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/784ab2ff-0096-4a1d-958d-734cdc1ead62-metrics-client-ca\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.283029 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.282969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-secret-telemeter-client\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.384101 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.384061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784ab2ff-0096-4a1d-958d-734cdc1ead62-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.384276 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.384109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.384276 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.384172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784ab2ff-0096-4a1d-958d-734cdc1ead62-serving-certs-ca-bundle\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.384276 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.384209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjq8k\" (UniqueName: \"kubernetes.io/projected/784ab2ff-0096-4a1d-958d-734cdc1ead62-kube-api-access-fjq8k\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.384276 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.384267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-telemeter-client-tls\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.384506 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.384320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/784ab2ff-0096-4a1d-958d-734cdc1ead62-metrics-client-ca\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.384506 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.384437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-secret-telemeter-client\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.384625 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.384508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-federate-client-tls\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.385006 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.384977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784ab2ff-0096-4a1d-958d-734cdc1ead62-serving-certs-ca-bundle\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.385374 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.385351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/784ab2ff-0096-4a1d-958d-734cdc1ead62-metrics-client-ca\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.385479 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.385394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784ab2ff-0096-4a1d-958d-734cdc1ead62-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.386780 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.386748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.386954 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.386937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-secret-telemeter-client\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.387028 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.387013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-federate-client-tls\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.387206 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.387187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/784ab2ff-0096-4a1d-958d-734cdc1ead62-telemeter-client-tls\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.392475 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.392456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjq8k\" (UniqueName: \"kubernetes.io/projected/784ab2ff-0096-4a1d-958d-734cdc1ead62-kube-api-access-fjq8k\") pod \"telemeter-client-77f855b89f-t6ps6\" (UID: \"784ab2ff-0096-4a1d-958d-734cdc1ead62\") " pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.499770 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.499740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6"
Jul 23 18:19:12.631927 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:12.631902    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77f855b89f-t6ps6"]
Jul 23 18:19:12.634401 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:19:12.634377    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784ab2ff_0096_4a1d_958d_734cdc1ead62.slice/crio-d0bdfd62e18eeeaa1dccb88d3385e624990d0b77ce982c15e0c2276b84e4c935 WatchSource:0}: Error finding container d0bdfd62e18eeeaa1dccb88d3385e624990d0b77ce982c15e0c2276b84e4c935: Status 404 returned error can't find the container with id d0bdfd62e18eeeaa1dccb88d3385e624990d0b77ce982c15e0c2276b84e4c935
Jul 23 18:19:13.623319 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:13.623270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6" event={"ID":"784ab2ff-0096-4a1d-958d-734cdc1ead62","Type":"ContainerStarted","Data":"d0bdfd62e18eeeaa1dccb88d3385e624990d0b77ce982c15e0c2276b84e4c935"}
Jul 23 18:19:15.634919 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:15.634882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6" event={"ID":"784ab2ff-0096-4a1d-958d-734cdc1ead62","Type":"ContainerStarted","Data":"46d2ac857a0e6cbf53b8117169106481e4bb5a7d3f8fb5cb097f0a0f33fd5e50"}
Jul 23 18:19:15.635275 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:15.634940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6" event={"ID":"784ab2ff-0096-4a1d-958d-734cdc1ead62","Type":"ContainerStarted","Data":"9184a05ae8ceaee59fe26d140dd546bcbc0fcb643e140e759e35bcedafcc3a49"}
Jul 23 18:19:15.635275 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:15.634960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6" event={"ID":"784ab2ff-0096-4a1d-958d-734cdc1ead62","Type":"ContainerStarted","Data":"13889b356591bebfb1c0b111c3964d6924d678f2d5267d54198aff5a83abb28a"}
Jul 23 18:19:15.658551 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:15.658503    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-77f855b89f-t6ps6" podStartSLOduration=1.624181069 podStartE2EDuration="3.658486424s" totalImagesPullingTime="2.034305355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:19:15.657007739 +0000 UTC m=+151.504698290" watchObservedRunningTime="2026-07-23 18:19:15.658486424 +0000 UTC m=+151.506176947"
Jul 23 18:19:36.866787 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:19:36.866704    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bd9f8558c-jc5g8"]
Jul 23 18:20:01.887302 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:01.887222    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bd9f8558c-jc5g8" podUID="8d7074c5-f4de-4bc8-8681-a3913f3156d5" containerName="console" containerID="cri-o://a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f" gracePeriod=15
Jul 23 18:20:02.132926 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.132900    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:20:02.216006 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.215976    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-config\") pod \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") "
Jul 23 18:20:02.216181 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216025    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-oauth-config\") pod \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") "
Jul 23 18:20:02.216181 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216046    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-oauth-serving-cert\") pod \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") "
Jul 23 18:20:02.216327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216170    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-trusted-ca-bundle\") pod \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") "
Jul 23 18:20:02.216327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216223    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-serving-cert\") pod \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") "
Jul 23 18:20:02.216327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216252    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7w68\" (UniqueName: \"kubernetes.io/projected/8d7074c5-f4de-4bc8-8681-a3913f3156d5-kube-api-access-h7w68\") pod \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") "
Jul 23 18:20:02.216327 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216312    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-service-ca\") pod \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\" (UID: \"8d7074c5-f4de-4bc8-8681-a3913f3156d5\") "
Jul 23 18:20:02.216531 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216494    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8d7074c5-f4de-4bc8-8681-a3913f3156d5" (UID: "8d7074c5-f4de-4bc8-8681-a3913f3156d5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:20:02.216587 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-config" (OuterVolumeSpecName: "console-config") pod "8d7074c5-f4de-4bc8-8681-a3913f3156d5" (UID: "8d7074c5-f4de-4bc8-8681-a3913f3156d5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:20:02.216647 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216606    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-config\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:20:02.216647 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216623    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-oauth-serving-cert\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:20:02.216755 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216618    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8d7074c5-f4de-4bc8-8681-a3913f3156d5" (UID: "8d7074c5-f4de-4bc8-8681-a3913f3156d5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:20:02.216813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.216788    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-service-ca" (OuterVolumeSpecName: "service-ca") pod "8d7074c5-f4de-4bc8-8681-a3913f3156d5" (UID: "8d7074c5-f4de-4bc8-8681-a3913f3156d5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:20:02.218521 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.218490    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7074c5-f4de-4bc8-8681-a3913f3156d5-kube-api-access-h7w68" (OuterVolumeSpecName: "kube-api-access-h7w68") pod "8d7074c5-f4de-4bc8-8681-a3913f3156d5" (UID: "8d7074c5-f4de-4bc8-8681-a3913f3156d5"). InnerVolumeSpecName "kube-api-access-h7w68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:20:02.218521 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.218501    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8d7074c5-f4de-4bc8-8681-a3913f3156d5" (UID: "8d7074c5-f4de-4bc8-8681-a3913f3156d5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:20:02.218657 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.218539    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8d7074c5-f4de-4bc8-8681-a3913f3156d5" (UID: "8d7074c5-f4de-4bc8-8681-a3913f3156d5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:20:02.317993 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.317956    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-oauth-config\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:20:02.317993 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.317986    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-trusted-ca-bundle\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:20:02.317993 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.317996    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d7074c5-f4de-4bc8-8681-a3913f3156d5-console-serving-cert\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:20:02.318221 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.318004    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7w68\" (UniqueName: \"kubernetes.io/projected/8d7074c5-f4de-4bc8-8681-a3913f3156d5-kube-api-access-h7w68\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:20:02.318221 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.318013    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d7074c5-f4de-4bc8-8681-a3913f3156d5-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:20:02.822703 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.822669    2561 generic.go:356] "Generic (PLEG): container finished" podID="8d7074c5-f4de-4bc8-8681-a3913f3156d5" containerID="a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f" exitCode=0
Jul 23 18:20:02.822703 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.822708    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd9f8558c-jc5g8" event={"ID":"8d7074c5-f4de-4bc8-8681-a3913f3156d5","Type":"ContainerDied","Data":"a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f"}
Jul 23 18:20:02.822904 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.822729    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bd9f8558c-jc5g8"
Jul 23 18:20:02.822904 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.822745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bd9f8558c-jc5g8" event={"ID":"8d7074c5-f4de-4bc8-8681-a3913f3156d5","Type":"ContainerDied","Data":"263ebd14472549128a7b0e69d9e6f07371171a369be2c0606b622486f315e353"}
Jul 23 18:20:02.822904 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.822764    2561 scope.go:117] "RemoveContainer" containerID="a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f"
Jul 23 18:20:02.831304 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.831271    2561 scope.go:117] "RemoveContainer" containerID="a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f"
Jul 23 18:20:02.831549 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:20:02.831526    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f\": container with ID starting with a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f not found: ID does not exist" containerID="a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f"
Jul 23 18:20:02.831644 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.831555    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f"} err="failed to get container status \"a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f\": rpc error: code = NotFound desc = could not find container \"a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f\": container with ID starting with a61043fdc92aaf2c18300c218b429d9998b68bd0aaba0a364206d7fc2d2fc82f not found: ID does not exist"
Jul 23 18:20:02.840865 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.840842    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bd9f8558c-jc5g8"]
Jul 23 18:20:02.844609 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:02.844587    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bd9f8558c-jc5g8"]
Jul 23 18:20:04.760328 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:04.760278    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d7074c5-f4de-4bc8-8681-a3913f3156d5" path="/var/lib/kubelet/pods/8d7074c5-f4de-4bc8-8681-a3913f3156d5/volumes"
Jul 23 18:20:57.690421 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.690385    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6f8vn"]
Jul 23 18:20:57.690978 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.690883    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d7074c5-f4de-4bc8-8681-a3913f3156d5" containerName="console"
Jul 23 18:20:57.690978 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.690902    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d7074c5-f4de-4bc8-8681-a3913f3156d5" containerName="console"
Jul 23 18:20:57.691095 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.691002    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d7074c5-f4de-4bc8-8681-a3913f3156d5" containerName="console"
Jul 23 18:20:57.695163 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.695142    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:20:57.698788 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.698763    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-44z2s\""
Jul 23 18:20:57.698895 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.698880    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 23 18:20:57.699771 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.699752    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 23 18:20:57.717885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.717861    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6f8vn"]
Jul 23 18:20:57.796420 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.796384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec51ac2b-5ea7-4008-9628-c5c747716d3a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6f8vn\" (UID: \"ec51ac2b-5ea7-4008-9628-c5c747716d3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:20:57.796565 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.796481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lgx\" (UniqueName: \"kubernetes.io/projected/ec51ac2b-5ea7-4008-9628-c5c747716d3a-kube-api-access-p2lgx\") pod \"cert-manager-webhook-6486ffd997-6f8vn\" (UID: \"ec51ac2b-5ea7-4008-9628-c5c747716d3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:20:57.897759 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.897728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lgx\" (UniqueName: \"kubernetes.io/projected/ec51ac2b-5ea7-4008-9628-c5c747716d3a-kube-api-access-p2lgx\") pod \"cert-manager-webhook-6486ffd997-6f8vn\" (UID: \"ec51ac2b-5ea7-4008-9628-c5c747716d3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:20:57.897936 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.897783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec51ac2b-5ea7-4008-9628-c5c747716d3a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6f8vn\" (UID: \"ec51ac2b-5ea7-4008-9628-c5c747716d3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:20:57.913514 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.913482    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lgx\" (UniqueName: \"kubernetes.io/projected/ec51ac2b-5ea7-4008-9628-c5c747716d3a-kube-api-access-p2lgx\") pod \"cert-manager-webhook-6486ffd997-6f8vn\" (UID: \"ec51ac2b-5ea7-4008-9628-c5c747716d3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:20:57.914551 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:57.914527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec51ac2b-5ea7-4008-9628-c5c747716d3a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6f8vn\" (UID: \"ec51ac2b-5ea7-4008-9628-c5c747716d3a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:20:58.023173 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:58.023134    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:20:58.149699 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:58.149625    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6f8vn"]
Jul 23 18:20:58.151990 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:20:58.151963    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec51ac2b_5ea7_4008_9628_c5c747716d3a.slice/crio-cf1a17c0be7ae326e4275d1c8fa34210f4673fb8820656495740e1985de03d7f WatchSource:0}: Error finding container cf1a17c0be7ae326e4275d1c8fa34210f4673fb8820656495740e1985de03d7f: Status 404 returned error can't find the container with id cf1a17c0be7ae326e4275d1c8fa34210f4673fb8820656495740e1985de03d7f
Jul 23 18:20:59.052143 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:20:59.052100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn" event={"ID":"ec51ac2b-5ea7-4008-9628-c5c747716d3a","Type":"ContainerStarted","Data":"cf1a17c0be7ae326e4275d1c8fa34210f4673fb8820656495740e1985de03d7f"}
Jul 23 18:21:02.068612 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:02.068575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn" event={"ID":"ec51ac2b-5ea7-4008-9628-c5c747716d3a","Type":"ContainerStarted","Data":"c23f3e4bb9277e52ff212f0f824d837184fa555310047d922f5703a158ac72a7"}
Jul 23 18:21:02.068983 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:02.068627    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:21:02.089134 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:02.089071    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn" podStartSLOduration=1.915004435 podStartE2EDuration="5.089051111s" totalImagesPullingTime="3.174046676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:02.087407859 +0000 UTC m=+257.935098386" watchObservedRunningTime="2026-07-23 18:21:02.089051111 +0000 UTC m=+257.936741640"
Jul 23 18:21:08.074060 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:08.073975    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6f8vn"
Jul 23 18:21:15.605835 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.605796    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"]
Jul 23 18:21:15.609222 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.609201    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"
Jul 23 18:21:15.611504 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.611479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 18:21:15.611620 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.611489    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 18:21:15.611620 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.611530    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kc8hb\""
Jul 23 18:21:15.618784 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.618761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"]
Jul 23 18:21:15.749757 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.749718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxt6\" (UniqueName: \"kubernetes.io/projected/1c253c3b-da1b-4ec4-a684-24964a39afc1-kube-api-access-vgxt6\") pod \"openshift-lws-operator-fd8ccff4c-gtgq2\" (UID: \"1c253c3b-da1b-4ec4-a684-24964a39afc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"
Jul 23 18:21:15.749914 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.749763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c253c3b-da1b-4ec4-a684-24964a39afc1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gtgq2\" (UID: \"1c253c3b-da1b-4ec4-a684-24964a39afc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"
Jul 23 18:21:15.850346 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.850314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxt6\" (UniqueName: \"kubernetes.io/projected/1c253c3b-da1b-4ec4-a684-24964a39afc1-kube-api-access-vgxt6\") pod \"openshift-lws-operator-fd8ccff4c-gtgq2\" (UID: \"1c253c3b-da1b-4ec4-a684-24964a39afc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"
Jul 23 18:21:15.850484 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.850351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c253c3b-da1b-4ec4-a684-24964a39afc1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gtgq2\" (UID: \"1c253c3b-da1b-4ec4-a684-24964a39afc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"
Jul 23 18:21:15.850682 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.850666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c253c3b-da1b-4ec4-a684-24964a39afc1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gtgq2\" (UID: \"1c253c3b-da1b-4ec4-a684-24964a39afc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"
Jul 23 18:21:15.858407 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.858355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxt6\" (UniqueName: \"kubernetes.io/projected/1c253c3b-da1b-4ec4-a684-24964a39afc1-kube-api-access-vgxt6\") pod \"openshift-lws-operator-fd8ccff4c-gtgq2\" (UID: \"1c253c3b-da1b-4ec4-a684-24964a39afc1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"
Jul 23 18:21:15.917663 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:15.917635    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"
Jul 23 18:21:16.040917 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:16.040894    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2"]
Jul 23 18:21:16.042993 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:21:16.042969    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c253c3b_da1b_4ec4_a684_24964a39afc1.slice/crio-c13118a5d4fb0074b9735d3273836918cf4cd9ee8e57c513a13dac4e684cf985 WatchSource:0}: Error finding container c13118a5d4fb0074b9735d3273836918cf4cd9ee8e57c513a13dac4e684cf985: Status 404 returned error can't find the container with id c13118a5d4fb0074b9735d3273836918cf4cd9ee8e57c513a13dac4e684cf985
Jul 23 18:21:16.122356 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:16.122253    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2" event={"ID":"1c253c3b-da1b-4ec4-a684-24964a39afc1","Type":"ContainerStarted","Data":"c13118a5d4fb0074b9735d3273836918cf4cd9ee8e57c513a13dac4e684cf985"}
Jul 23 18:21:19.138466 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:19.138376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2" event={"ID":"1c253c3b-da1b-4ec4-a684-24964a39afc1","Type":"ContainerStarted","Data":"7d5ed16ef8406a97f6ed59fda1f30b3cb2eb6462c5eb5f55c5b931e1af3cbb68"}
Jul 23 18:21:19.155130 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:19.155078    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gtgq2" podStartSLOduration=1.417259134 podStartE2EDuration="4.155063234s" totalImagesPullingTime="2.7378041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:19.153263809 +0000 UTC m=+275.000954354" watchObservedRunningTime="2026-07-23 18:21:19.155063234 +0000 UTC m=+275.002753761"
Jul 23 18:21:35.392480 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.392451    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"]
Jul 23 18:21:35.395015 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.394993    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.399088 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.399065    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 18:21:35.400433 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.400408    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 18:21:35.400636 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.400600    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 18:21:35.400751 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.400674    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-565dp\""
Jul 23 18:21:35.400751 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.400703    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 18:21:35.441161 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.441133    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"]
Jul 23 18:21:35.510760 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.510721    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45rh\" (UniqueName: \"kubernetes.io/projected/37dedd14-286c-4b6e-a696-398b5b875b8e-kube-api-access-l45rh\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.510760 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.510761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/37dedd14-286c-4b6e-a696-398b5b875b8e-manifests\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.510995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.510859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37dedd14-286c-4b6e-a696-398b5b875b8e-tmp\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.510995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.510902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37dedd14-286c-4b6e-a696-398b5b875b8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.510995 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.510977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/37dedd14-286c-4b6e-a696-398b5b875b8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.612124 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.612085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37dedd14-286c-4b6e-a696-398b5b875b8e-tmp\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.612348 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.612139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37dedd14-286c-4b6e-a696-398b5b875b8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.612348 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.612180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/37dedd14-286c-4b6e-a696-398b5b875b8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.612348 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.612217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45rh\" (UniqueName: \"kubernetes.io/projected/37dedd14-286c-4b6e-a696-398b5b875b8e-kube-api-access-l45rh\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.612348 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.612240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/37dedd14-286c-4b6e-a696-398b5b875b8e-manifests\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.612621 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.612592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/37dedd14-286c-4b6e-a696-398b5b875b8e-manifests\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.614438 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.614413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37dedd14-286c-4b6e-a696-398b5b875b8e-tmp\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.614581 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.614563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/37dedd14-286c-4b6e-a696-398b5b875b8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.614655 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.614637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37dedd14-286c-4b6e-a696-398b5b875b8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.619938 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.619919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45rh\" (UniqueName: \"kubernetes.io/projected/37dedd14-286c-4b6e-a696-398b5b875b8e-kube-api-access-l45rh\") pod \"opendatahub-operator-controller-manager-77bdbd87db-hwlbr\" (UID: \"37dedd14-286c-4b6e-a696-398b5b875b8e\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.705516 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.705484    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:35.840171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:35.840113    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"]
Jul 23 18:21:35.848694 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:21:35.848659    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37dedd14_286c_4b6e_a696_398b5b875b8e.slice/crio-ffb5f7c0e0e8c0b406a52a539cfc7eac2dca93cfe089d261246542b9ad59d2b8 WatchSource:0}: Error finding container ffb5f7c0e0e8c0b406a52a539cfc7eac2dca93cfe089d261246542b9ad59d2b8: Status 404 returned error can't find the container with id ffb5f7c0e0e8c0b406a52a539cfc7eac2dca93cfe089d261246542b9ad59d2b8
Jul 23 18:21:36.209117 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:36.209079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr" event={"ID":"37dedd14-286c-4b6e-a696-398b5b875b8e","Type":"ContainerStarted","Data":"ffb5f7c0e0e8c0b406a52a539cfc7eac2dca93cfe089d261246542b9ad59d2b8"}
Jul 23 18:21:39.227355 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:39.227318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr" event={"ID":"37dedd14-286c-4b6e-a696-398b5b875b8e","Type":"ContainerStarted","Data":"ac547a1d3e4ece03392dbf4731a2624b0d7a60032490f514a3fc420b3b8dac6a"}
Jul 23 18:21:40.233716 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:40.233681    2561 generic.go:356] "Generic (PLEG): container finished" podID="37dedd14-286c-4b6e-a696-398b5b875b8e" containerID="ac547a1d3e4ece03392dbf4731a2624b0d7a60032490f514a3fc420b3b8dac6a" exitCode=0
Jul 23 18:21:40.234165 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:40.233760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr" event={"ID":"37dedd14-286c-4b6e-a696-398b5b875b8e","Type":"ContainerDied","Data":"ac547a1d3e4ece03392dbf4731a2624b0d7a60032490f514a3fc420b3b8dac6a"}
Jul 23 18:21:41.241220 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:41.241184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr" event={"ID":"37dedd14-286c-4b6e-a696-398b5b875b8e","Type":"ContainerStarted","Data":"40f51f77f6ff210d68bda89310085e1e30691157dca9bb81fac5e71982b62416"}
Jul 23 18:21:41.241714 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:41.241312    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:41.263784 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:41.263737    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr" podStartSLOduration=2.006676977 podStartE2EDuration="6.263721803s" totalImagesPullingTime="3.253594451s" totalInitContainerRuntime="1.003450375s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:41.261514403 +0000 UTC m=+297.109204931" watchObservedRunningTime="2026-07-23 18:21:41.263721803 +0000 UTC m=+297.111412330"
Jul 23 18:21:44.567227 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:44.567202    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:21:44.567749 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:44.567732    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:21:52.248815 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:52.248786    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-hwlbr"
Jul 23 18:21:54.599365 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.599327    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"]
Jul 23 18:21:54.601868 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.601809    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.605394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.605364    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 18:21:54.612908 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.609561    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6gqr7\""
Jul 23 18:21:54.612908 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.609861    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 18:21:54.612908 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.609869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 18:21:54.618739 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.618711    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"]
Jul 23 18:21:54.683079 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.683045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qprr\" (UniqueName: \"kubernetes.io/projected/ce843df6-358e-4733-a1b0-f7b9a0d0f716-kube-api-access-7qprr\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.683250 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.683122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce843df6-358e-4733-a1b0-f7b9a0d0f716-metrics-cert\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.683250 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.683155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ce843df6-358e-4733-a1b0-f7b9a0d0f716-manager-config\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.683250 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.683202    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce843df6-358e-4733-a1b0-f7b9a0d0f716-cert\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.784400 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.784367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce843df6-358e-4733-a1b0-f7b9a0d0f716-metrics-cert\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.784400 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.784402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ce843df6-358e-4733-a1b0-f7b9a0d0f716-manager-config\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.784616 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.784445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce843df6-358e-4733-a1b0-f7b9a0d0f716-cert\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.784616 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.784564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qprr\" (UniqueName: \"kubernetes.io/projected/ce843df6-358e-4733-a1b0-f7b9a0d0f716-kube-api-access-7qprr\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.785163 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.785136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ce843df6-358e-4733-a1b0-f7b9a0d0f716-manager-config\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.786817 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.786789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce843df6-358e-4733-a1b0-f7b9a0d0f716-metrics-cert\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.786913 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.786837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce843df6-358e-4733-a1b0-f7b9a0d0f716-cert\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.811832 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.811802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qprr\" (UniqueName: \"kubernetes.io/projected/ce843df6-358e-4733-a1b0-f7b9a0d0f716-kube-api-access-7qprr\") pod \"lws-controller-manager-778f8cb4df-jcp7l\" (UID: \"ce843df6-358e-4733-a1b0-f7b9a0d0f716\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:54.919598 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:54.919511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:55.044096 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.044056    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"]
Jul 23 18:21:55.047269 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.047242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.052339 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.052321    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 23 18:21:55.052695 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.052677    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 23 18:21:55.053131 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.053111    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 18:21:55.053131 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.053122    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 18:21:55.053276 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.053160    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-j4n9q\""
Jul 23 18:21:55.061043 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.061020    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"]
Jul 23 18:21:55.075402 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.075377    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"]
Jul 23 18:21:55.078582 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:21:55.078561    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce843df6_358e_4733_a1b0_f7b9a0d0f716.slice/crio-04034901fce37255fcf4cbec413c4b5353981d374bf440868d441c986f30991e WatchSource:0}: Error finding container 04034901fce37255fcf4cbec413c4b5353981d374bf440868d441c986f30991e: Status 404 returned error can't find the container with id 04034901fce37255fcf4cbec413c4b5353981d374bf440868d441c986f30991e
Jul 23 18:21:55.080601 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.080578    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:21:55.187195 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.187113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f75f153-7e86-4dc0-ad37-024ac81f3fba-tmp\") pod \"kube-auth-proxy-6b75b7b5f6-598qm\" (UID: \"2f75f153-7e86-4dc0-ad37-024ac81f3fba\") " pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.187195 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.187183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f75f153-7e86-4dc0-ad37-024ac81f3fba-tls-certs\") pod \"kube-auth-proxy-6b75b7b5f6-598qm\" (UID: \"2f75f153-7e86-4dc0-ad37-024ac81f3fba\") " pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.187394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.187216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpqt\" (UniqueName: \"kubernetes.io/projected/2f75f153-7e86-4dc0-ad37-024ac81f3fba-kube-api-access-4gpqt\") pod \"kube-auth-proxy-6b75b7b5f6-598qm\" (UID: \"2f75f153-7e86-4dc0-ad37-024ac81f3fba\") " pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.288216 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.288179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f75f153-7e86-4dc0-ad37-024ac81f3fba-tls-certs\") pod \"kube-auth-proxy-6b75b7b5f6-598qm\" (UID: \"2f75f153-7e86-4dc0-ad37-024ac81f3fba\") " pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.288216 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.288227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpqt\" (UniqueName: \"kubernetes.io/projected/2f75f153-7e86-4dc0-ad37-024ac81f3fba-kube-api-access-4gpqt\") pod \"kube-auth-proxy-6b75b7b5f6-598qm\" (UID: \"2f75f153-7e86-4dc0-ad37-024ac81f3fba\") " pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.288487 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.288303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f75f153-7e86-4dc0-ad37-024ac81f3fba-tmp\") pod \"kube-auth-proxy-6b75b7b5f6-598qm\" (UID: \"2f75f153-7e86-4dc0-ad37-024ac81f3fba\") " pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.290528 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.290500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f75f153-7e86-4dc0-ad37-024ac81f3fba-tmp\") pod \"kube-auth-proxy-6b75b7b5f6-598qm\" (UID: \"2f75f153-7e86-4dc0-ad37-024ac81f3fba\") " pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.290724 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.290706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f75f153-7e86-4dc0-ad37-024ac81f3fba-tls-certs\") pod \"kube-auth-proxy-6b75b7b5f6-598qm\" (UID: \"2f75f153-7e86-4dc0-ad37-024ac81f3fba\") " pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.304445 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.304413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l" event={"ID":"ce843df6-358e-4733-a1b0-f7b9a0d0f716","Type":"ContainerStarted","Data":"04034901fce37255fcf4cbec413c4b5353981d374bf440868d441c986f30991e"}
Jul 23 18:21:55.309302 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.309260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpqt\" (UniqueName: \"kubernetes.io/projected/2f75f153-7e86-4dc0-ad37-024ac81f3fba-kube-api-access-4gpqt\") pod \"kube-auth-proxy-6b75b7b5f6-598qm\" (UID: \"2f75f153-7e86-4dc0-ad37-024ac81f3fba\") " pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.359582 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.359547    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"
Jul 23 18:21:55.513278 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:55.513254    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm"]
Jul 23 18:21:55.515711 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:21:55.515677    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f75f153_7e86_4dc0_ad37_024ac81f3fba.slice/crio-1379085cee48ad51f099443078e926d96e575506541b23e1e8a2e095f94ee063 WatchSource:0}: Error finding container 1379085cee48ad51f099443078e926d96e575506541b23e1e8a2e095f94ee063: Status 404 returned error can't find the container with id 1379085cee48ad51f099443078e926d96e575506541b23e1e8a2e095f94ee063
Jul 23 18:21:56.313416 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:56.313380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm" event={"ID":"2f75f153-7e86-4dc0-ad37-024ac81f3fba","Type":"ContainerStarted","Data":"1379085cee48ad51f099443078e926d96e575506541b23e1e8a2e095f94ee063"}
Jul 23 18:21:57.322049 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:57.321956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l" event={"ID":"ce843df6-358e-4733-a1b0-f7b9a0d0f716","Type":"ContainerStarted","Data":"32390cc99b22db14bdf76542c49d450282caddfa9007d2ee74bcfa49fcf3d427"}
Jul 23 18:21:57.322496 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:57.322104    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:21:57.344567 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:57.344470    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l" podStartSLOduration=1.2168181200000001 podStartE2EDuration="3.344447289s" totalImagesPullingTime="2.127629169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:57.343479904 +0000 UTC m=+313.191170433" watchObservedRunningTime="2026-07-23 18:21:57.344447289 +0000 UTC m=+313.192137817"
Jul 23 18:21:59.333515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:59.333470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm" event={"ID":"2f75f153-7e86-4dc0-ad37-024ac81f3fba","Type":"ContainerStarted","Data":"dfc1daab3ef3665007b0d82f3b37150b689c1d42bba4e79e30612c67a674619a"}
Jul 23 18:21:59.351687 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:21:59.351625    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b75b7b5f6-598qm" podStartSLOduration=1.500043975 podStartE2EDuration="4.351604855s" totalImagesPullingTime="2.85156088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:59.348538925 +0000 UTC m=+315.196229453" watchObservedRunningTime="2026-07-23 18:21:59.351604855 +0000 UTC m=+315.199295383"
Jul 23 18:22:08.330370 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:22:08.330335    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-jcp7l"
Jul 23 18:23:32.319682 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.319651    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"]
Jul 23 18:23:32.323100 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.323079    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"
Jul 23 18:23:32.326035 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.326010    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 18:23:32.326153 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.326034    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-j8qgd\""
Jul 23 18:23:32.326153 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.326066    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 23 18:23:32.326644 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.326628    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 18:23:32.333557 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.333534    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"]
Jul 23 18:23:32.447895 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.447858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6p95\" (UniqueName: \"kubernetes.io/projected/3e31c93f-895b-4ac0-884d-bd034db9b08f-kube-api-access-f6p95\") pod \"dns-operator-controller-manager-864dd78c4c-vj62c\" (UID: \"3e31c93f-895b-4ac0-884d-bd034db9b08f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"
Jul 23 18:23:32.549161 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.549126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6p95\" (UniqueName: \"kubernetes.io/projected/3e31c93f-895b-4ac0-884d-bd034db9b08f-kube-api-access-f6p95\") pod \"dns-operator-controller-manager-864dd78c4c-vj62c\" (UID: \"3e31c93f-895b-4ac0-884d-bd034db9b08f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"
Jul 23 18:23:32.557907 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.557875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6p95\" (UniqueName: \"kubernetes.io/projected/3e31c93f-895b-4ac0-884d-bd034db9b08f-kube-api-access-f6p95\") pod \"dns-operator-controller-manager-864dd78c4c-vj62c\" (UID: \"3e31c93f-895b-4ac0-884d-bd034db9b08f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"
Jul 23 18:23:32.634463 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.634378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"
Jul 23 18:23:32.766742 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:32.766719    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"]
Jul 23 18:23:32.769357 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:23:32.769331    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e31c93f_895b_4ac0_884d_bd034db9b08f.slice/crio-28d93d9297412db9b7feb0c75fa3d9f40971d0c4e311f305f9ab2278e6a5e22e WatchSource:0}: Error finding container 28d93d9297412db9b7feb0c75fa3d9f40971d0c4e311f305f9ab2278e6a5e22e: Status 404 returned error can't find the container with id 28d93d9297412db9b7feb0c75fa3d9f40971d0c4e311f305f9ab2278e6a5e22e
Jul 23 18:23:33.763246 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:33.763214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c" event={"ID":"3e31c93f-895b-4ac0-884d-bd034db9b08f","Type":"ContainerStarted","Data":"28d93d9297412db9b7feb0c75fa3d9f40971d0c4e311f305f9ab2278e6a5e22e"}
Jul 23 18:23:35.774059 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:35.774015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c" event={"ID":"3e31c93f-895b-4ac0-884d-bd034db9b08f","Type":"ContainerStarted","Data":"3b57878da4ade17e9b3a47d37b84a727a94aa475887f7a081857cad1e6ad6217"}
Jul 23 18:23:35.774476 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:35.774164    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"
Jul 23 18:23:35.793168 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:35.793123    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c" podStartSLOduration=1.438412744 podStartE2EDuration="3.793108613s" totalImagesPullingTime="2.354695869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:23:35.792139606 +0000 UTC m=+411.639830133" watchObservedRunningTime="2026-07-23 18:23:35.793108613 +0000 UTC m=+411.640799140"
Jul 23 18:23:43.287484 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.287447    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"]
Jul 23 18:23:43.290787 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.290762    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"
Jul 23 18:23:43.293357 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.293335    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-m5snv\""
Jul 23 18:23:43.303999 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.303973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"]
Jul 23 18:23:43.450196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.450163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6d7p\" (UniqueName: \"kubernetes.io/projected/39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1-kube-api-access-j6d7p\") pod \"limitador-operator-controller-manager-859d7b78bd-zg825\" (UID: \"39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"
Jul 23 18:23:43.551703 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.551610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6d7p\" (UniqueName: \"kubernetes.io/projected/39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1-kube-api-access-j6d7p\") pod \"limitador-operator-controller-manager-859d7b78bd-zg825\" (UID: \"39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"
Jul 23 18:23:43.560174 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.560145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6d7p\" (UniqueName: \"kubernetes.io/projected/39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1-kube-api-access-j6d7p\") pod \"limitador-operator-controller-manager-859d7b78bd-zg825\" (UID: \"39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"
Jul 23 18:23:43.602013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.601979    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"
Jul 23 18:23:43.733534 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.733502    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"]
Jul 23 18:23:43.736069 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:23:43.736040    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ad9f6f_e586_4e98_bf1b_63bfc45cd8d1.slice/crio-6b0d7be1e7ebfc00d8eb0a594c468cfe9edc8c8612ed4600fecf96af96cbbee5 WatchSource:0}: Error finding container 6b0d7be1e7ebfc00d8eb0a594c468cfe9edc8c8612ed4600fecf96af96cbbee5: Status 404 returned error can't find the container with id 6b0d7be1e7ebfc00d8eb0a594c468cfe9edc8c8612ed4600fecf96af96cbbee5
Jul 23 18:23:43.809603 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:43.809523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825" event={"ID":"39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1","Type":"ContainerStarted","Data":"6b0d7be1e7ebfc00d8eb0a594c468cfe9edc8c8612ed4600fecf96af96cbbee5"}
Jul 23 18:23:45.821748 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:45.821706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825" event={"ID":"39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1","Type":"ContainerStarted","Data":"22e1a42a5908355fbe280c4c9ff904b924f08bac546d2fac01ce946393828790"}
Jul 23 18:23:45.822149 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:45.821833    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"
Jul 23 18:23:45.844958 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:45.841468    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825" podStartSLOduration=1.240415764 podStartE2EDuration="2.841447604s" totalImagesPullingTime="1.60103184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:23:45.838627829 +0000 UTC m=+421.686318356" watchObservedRunningTime="2026-07-23 18:23:45.841447604 +0000 UTC m=+421.689138133"
Jul 23 18:23:46.780680 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:46.780650    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vj62c"
Jul 23 18:23:56.828962 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:23:56.828929    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zg825"
Jul 23 18:24:17.103086 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.103048    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:24:17.107474 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.107426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:24:17.110934 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.110911    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w6tlp\""
Jul 23 18:24:17.111075 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.110913    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 18:24:17.121056 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.121034    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:24:17.166547 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.166510    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:24:17.258102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.258068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b473aed2-ddf8-490b-9ef3-4993d9df05f6-config-file\") pod \"limitador-limitador-54d6d988d5-qplmq\" (UID: \"b473aed2-ddf8-490b-9ef3-4993d9df05f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:24:17.258261 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.258132    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8pz\" (UniqueName: \"kubernetes.io/projected/b473aed2-ddf8-490b-9ef3-4993d9df05f6-kube-api-access-gx8pz\") pod \"limitador-limitador-54d6d988d5-qplmq\" (UID: \"b473aed2-ddf8-490b-9ef3-4993d9df05f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:24:17.359499 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.359413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b473aed2-ddf8-490b-9ef3-4993d9df05f6-config-file\") pod \"limitador-limitador-54d6d988d5-qplmq\" (UID: \"b473aed2-ddf8-490b-9ef3-4993d9df05f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:24:17.359499 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.359480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8pz\" (UniqueName: \"kubernetes.io/projected/b473aed2-ddf8-490b-9ef3-4993d9df05f6-kube-api-access-gx8pz\") pod \"limitador-limitador-54d6d988d5-qplmq\" (UID: \"b473aed2-ddf8-490b-9ef3-4993d9df05f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:24:17.360150 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.360119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b473aed2-ddf8-490b-9ef3-4993d9df05f6-config-file\") pod \"limitador-limitador-54d6d988d5-qplmq\" (UID: \"b473aed2-ddf8-490b-9ef3-4993d9df05f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:24:17.368498 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.368474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8pz\" (UniqueName: \"kubernetes.io/projected/b473aed2-ddf8-490b-9ef3-4993d9df05f6-kube-api-access-gx8pz\") pod \"limitador-limitador-54d6d988d5-qplmq\" (UID: \"b473aed2-ddf8-490b-9ef3-4993d9df05f6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:24:17.418479 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.418440    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:24:17.549969 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.549938    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:24:17.553464 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:24:17.553436    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb473aed2_ddf8_490b_9ef3_4993d9df05f6.slice/crio-46ffe305b0f64fe5862dd72d72350f0a9a0dd5aa57ea9eddfeee7077ad043a77 WatchSource:0}: Error finding container 46ffe305b0f64fe5862dd72d72350f0a9a0dd5aa57ea9eddfeee7077ad043a77: Status 404 returned error can't find the container with id 46ffe305b0f64fe5862dd72d72350f0a9a0dd5aa57ea9eddfeee7077ad043a77
Jul 23 18:24:17.977509 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:17.977432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq" event={"ID":"b473aed2-ddf8-490b-9ef3-4993d9df05f6","Type":"ContainerStarted","Data":"46ffe305b0f64fe5862dd72d72350f0a9a0dd5aa57ea9eddfeee7077ad043a77"}
Jul 23 18:24:20.994122 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:20.994086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq" event={"ID":"b473aed2-ddf8-490b-9ef3-4993d9df05f6","Type":"ContainerStarted","Data":"d31f44eefd02460ca3b1eb52b2d276d339b37248c580c5e679d6890eec4afc83"}
Jul 23 18:24:20.994575 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:20.994162    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:24:21.011988 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:21.011936    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq" podStartSLOduration=1.585608341 podStartE2EDuration="4.011916825s" totalImagesPullingTime="2.426308484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:24:21.010748072 +0000 UTC m=+456.858438610" watchObservedRunningTime="2026-07-23 18:24:21.011916825 +0000 UTC m=+456.859607355"
Jul 23 18:24:32.000198 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:24:32.000164    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-qplmq"
Jul 23 18:25:34.940266 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:25:34.940230    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:26:05.590350 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:26:05.590250    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:26:40.587811 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:26:40.587777    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:26:44.595765 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:26:44.595726    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:26:44.600520 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:26:44.600495    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:26:48.791002 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:26:48.790966    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:26:59.393938 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:26:59.393900    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:27:24.197874 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:27:24.197842    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:27:35.691254 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:27:35.691213    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:28:48.007305 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:28:48.007206    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:28:57.699230 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:28:57.699197    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:29:08.012626 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:29:08.012592    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:30:51.879041 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:30:51.879000    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:31:00.918638 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:31:00.918594    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:31:34.290007 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:31:34.289969    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:31:43.685331 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:31:43.685209    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:31:44.625262 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:31:44.625225    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:31:44.630532 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:31:44.630512    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:31:53.985516 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:31:53.985480    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:31:59.480171 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:31:59.480136    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:32:09.986646 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:32:09.986602    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:32:19.503760 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:32:19.503723    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:32:31.884423 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:32:31.884386    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:32:41.679114 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:32:41.679076    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:33:03.593441 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:33:03.593405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:33:15.588240 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:33:15.588203    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:33:40.394413 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:33:40.394381    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:33:51.085666 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:33:51.085625    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:34:06.087801 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:34:06.087766    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:34:15.889512 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:34:15.889479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:34:25.380355 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:34:25.380310    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:34:35.690920 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:34:35.690883    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:34:44.983271 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:34:44.983180    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:36:31.977701 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:36:31.977664    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:36:43.297078 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:36:43.297041    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:36:44.654637 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:36:44.654612    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:36:44.660820 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:36:44.660797    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:36:52.277817 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:36:52.277779    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:37:04.012338 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:37:04.012297    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:37:14.885747 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:37:14.885712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:37:25.983686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:37:25.983648    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:37:35.288050 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:37:35.288005    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:37:46.688381 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:37:46.688296    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:37:56.192653 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:37:56.192616    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:38:05.616344 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:38:05.616304    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:38:16.578854 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:38:16.578819    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:38:38.096832 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:38:38.096802    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:38:47.110264 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:38:47.110230    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:39:02.582230 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:39:02.582179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:39:32.301653 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:39:32.301616    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:39:51.879703 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:39:51.879671    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:40:05.088054 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:40:05.088012    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:40:14.278458 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:40:14.278421    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:40:27.197905 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:40:27.197871    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:40:41.782797 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:40:41.782719    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:40:50.496357 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:40:50.496318    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:41:04.287020 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:41:04.286979    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:41:13.685524 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:41:13.685487    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:41:27.085913 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:41:27.085876    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:41:36.091752 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:41:36.091715    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:41:44.684936 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:41:44.684910    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:41:44.696675 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:41:44.696654    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:41:45.896988 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:41:45.896957    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:41:55.814991 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:41:55.814953    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:42:09.010662 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:42:09.010568    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:42:17.116521 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:42:17.116489    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:42:28.095020 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:42:28.094983    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:42:40.183535 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:42:40.183498    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:42:50.887688 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:42:50.887653    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:43:05.087147 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:05.087112    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:43:17.803706 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:17.803671    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:43:31.785860 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:31.785824    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:43:51.914343 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:51.914253    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"]
Jul 23 18:43:51.917514 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:51.917490    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"
Jul 23 18:43:51.920150 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:51.920121    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:43:51.920267 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:51.920132    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:43:51.925348 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:51.925020    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"]
Jul 23 18:43:52.040773 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:52.040736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7b466e-7014-4be6-8311-143faada6ece-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx\" (UID: \"5e7b466e-7014-4be6-8311-143faada6ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"
Jul 23 18:43:52.141515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:52.141474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7b466e-7014-4be6-8311-143faada6ece-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx\" (UID: \"5e7b466e-7014-4be6-8311-143faada6ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"
Jul 23 18:43:52.142130 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:52.142112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7b466e-7014-4be6-8311-143faada6ece-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx\" (UID: \"5e7b466e-7014-4be6-8311-143faada6ece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"
Jul 23 18:43:52.229585 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:52.229539    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"
Jul 23 18:43:52.357104 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:52.357060    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"]
Jul 23 18:43:52.359432 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:43:52.359406    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7b466e_7014_4be6_8311_143faada6ece.slice/crio-257a27eae524b6f8863404aa44d83768430d685d5dc7bda7d0412ba2fe8b1c36 WatchSource:0}: Error finding container 257a27eae524b6f8863404aa44d83768430d685d5dc7bda7d0412ba2fe8b1c36: Status 404 returned error can't find the container with id 257a27eae524b6f8863404aa44d83768430d685d5dc7bda7d0412ba2fe8b1c36
Jul 23 18:43:52.361198 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:52.361182    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:43:52.485411 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:52.485323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx" event={"ID":"5e7b466e-7014-4be6-8311-143faada6ece","Type":"ContainerStarted","Data":"257a27eae524b6f8863404aa44d83768430d685d5dc7bda7d0412ba2fe8b1c36"}
Jul 23 18:43:56.506676 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:56.506638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx" event={"ID":"5e7b466e-7014-4be6-8311-143faada6ece","Type":"ContainerStarted","Data":"ff1eab16b8d5033a72fea0576e9eb3eca9afe2df016b89f6b1123ad4aa1f9af7"}
Jul 23 18:43:56.526228 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:56.526183    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx" podStartSLOduration=2.002446141 podStartE2EDuration="5.526169558s" totalImagesPullingTime="3.523723417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:43:56.524399593 +0000 UTC m=+1632.372090151" watchObservedRunningTime="2026-07-23 18:43:56.526169558 +0000 UTC m=+1632.373860123"
Jul 23 18:43:57.512332 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:57.512279    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e7b466e-7014-4be6-8311-143faada6ece" containerID="ff1eab16b8d5033a72fea0576e9eb3eca9afe2df016b89f6b1123ad4aa1f9af7" exitCode=0
Jul 23 18:43:57.512796 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:57.512386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx" event={"ID":"5e7b466e-7014-4be6-8311-143faada6ece","Type":"ContainerDied","Data":"ff1eab16b8d5033a72fea0576e9eb3eca9afe2df016b89f6b1123ad4aa1f9af7"}
Jul 23 18:43:58.643094 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:58.643067    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"
Jul 23 18:43:58.700313 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:58.700257    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7b466e-7014-4be6-8311-143faada6ece-maas-api-service-ca\") pod \"5e7b466e-7014-4be6-8311-143faada6ece\" (UID: \"5e7b466e-7014-4be6-8311-143faada6ece\") "
Jul 23 18:43:58.700650 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:58.700626    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7b466e-7014-4be6-8311-143faada6ece-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e7b466e-7014-4be6-8311-143faada6ece" (UID: "5e7b466e-7014-4be6-8311-143faada6ece"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:43:58.801009 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:58.800978    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7b466e-7014-4be6-8311-143faada6ece-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:43:59.523806 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:59.523776    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"
Jul 23 18:43:59.523966 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:59.523803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx" event={"ID":"5e7b466e-7014-4be6-8311-143faada6ece","Type":"ContainerDied","Data":"257a27eae524b6f8863404aa44d83768430d685d5dc7bda7d0412ba2fe8b1c36"}
Jul 23 18:43:59.523966 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:43:59.523867    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="257a27eae524b6f8863404aa44d83768430d685d5dc7bda7d0412ba2fe8b1c36"
Jul 23 18:44:05.685651 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:05.685615    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:44:07.018440 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:07.018403    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"]
Jul 23 18:44:07.021829 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:07.021799    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9f5b583d-pgnvx"]
Jul 23 18:44:08.760663 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:08.760631    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e7b466e-7014-4be6-8311-143faada6ece" path="/var/lib/kubelet/pods/5e7b466e-7014-4be6-8311-143faada6ece/volumes"
Jul 23 18:44:16.903600 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:16.903567    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:44:35.756205 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.756168    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"]
Jul 23 18:44:35.756654 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.756620    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e7b466e-7014-4be6-8311-143faada6ece" containerName="revoke-keys"
Jul 23 18:44:35.756654 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.756634    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e7b466e-7014-4be6-8311-143faada6ece" containerName="revoke-keys"
Jul 23 18:44:35.756742 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.756690    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e7b466e-7014-4be6-8311-143faada6ece" containerName="revoke-keys"
Jul 23 18:44:35.759701 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.759685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"
Jul 23 18:44:35.761873 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.761849    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:44:35.762079 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.762061    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:44:35.766231 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.766210    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"]
Jul 23 18:44:35.820989 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.820954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfcb847f-b66d-4a4c-abf1-60848315d99a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6\" (UID: \"cfcb847f-b66d-4a4c-abf1-60848315d99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"
Jul 23 18:44:35.922375 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.922346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfcb847f-b66d-4a4c-abf1-60848315d99a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6\" (UID: \"cfcb847f-b66d-4a4c-abf1-60848315d99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"
Jul 23 18:44:35.922940 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:35.922919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfcb847f-b66d-4a4c-abf1-60848315d99a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6\" (UID: \"cfcb847f-b66d-4a4c-abf1-60848315d99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"
Jul 23 18:44:36.071380 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:36.071275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"
Jul 23 18:44:36.197393 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:36.197360    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"]
Jul 23 18:44:36.200759 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:44:36.200735    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcb847f_b66d_4a4c_abf1_60848315d99a.slice/crio-daee145ce88a1f43948723a1fda9645cf646bf35dd2679d0416daf5dd70ce3ef WatchSource:0}: Error finding container daee145ce88a1f43948723a1fda9645cf646bf35dd2679d0416daf5dd70ce3ef: Status 404 returned error can't find the container with id daee145ce88a1f43948723a1fda9645cf646bf35dd2679d0416daf5dd70ce3ef
Jul 23 18:44:36.703230 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:36.703192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6" event={"ID":"cfcb847f-b66d-4a4c-abf1-60848315d99a","Type":"ContainerStarted","Data":"c30ba66678ccb2881350f5ced9e85478a6e74ae9677edb23640df5f0f81e40ed"}
Jul 23 18:44:36.703399 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:36.703247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6" event={"ID":"cfcb847f-b66d-4a4c-abf1-60848315d99a","Type":"ContainerStarted","Data":"daee145ce88a1f43948723a1fda9645cf646bf35dd2679d0416daf5dd70ce3ef"}
Jul 23 18:44:36.719584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:36.719538    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6" podStartSLOduration=1.7195237460000001 podStartE2EDuration="1.719523746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:44:36.717704555 +0000 UTC m=+1672.565395079" watchObservedRunningTime="2026-07-23 18:44:36.719523746 +0000 UTC m=+1672.567214273"
Jul 23 18:44:37.709352 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:37.709319    2561 generic.go:356] "Generic (PLEG): container finished" podID="cfcb847f-b66d-4a4c-abf1-60848315d99a" containerID="c30ba66678ccb2881350f5ced9e85478a6e74ae9677edb23640df5f0f81e40ed" exitCode=0
Jul 23 18:44:37.709720 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:37.709399    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6" event={"ID":"cfcb847f-b66d-4a4c-abf1-60848315d99a","Type":"ContainerDied","Data":"c30ba66678ccb2881350f5ced9e85478a6e74ae9677edb23640df5f0f81e40ed"}
Jul 23 18:44:38.841196 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:38.841173    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"
Jul 23 18:44:38.950225 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:38.950191    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfcb847f-b66d-4a4c-abf1-60848315d99a-maas-api-service-ca\") pod \"cfcb847f-b66d-4a4c-abf1-60848315d99a\" (UID: \"cfcb847f-b66d-4a4c-abf1-60848315d99a\") "
Jul 23 18:44:38.950559 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:38.950536    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfcb847f-b66d-4a4c-abf1-60848315d99a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfcb847f-b66d-4a4c-abf1-60848315d99a" (UID: "cfcb847f-b66d-4a4c-abf1-60848315d99a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:44:39.051382 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:39.051316    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfcb847f-b66d-4a4c-abf1-60848315d99a-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:44:39.719762 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:39.719736    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"
Jul 23 18:44:39.719930 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:39.719767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6" event={"ID":"cfcb847f-b66d-4a4c-abf1-60848315d99a","Type":"ContainerDied","Data":"daee145ce88a1f43948723a1fda9645cf646bf35dd2679d0416daf5dd70ce3ef"}
Jul 23 18:44:39.719930 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:39.719807    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daee145ce88a1f43948723a1fda9645cf646bf35dd2679d0416daf5dd70ce3ef"
Jul 23 18:44:49.286181 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:49.286138    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:44:50.843040 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:50.843005    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"]
Jul 23 18:44:50.846853 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:50.846829    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4dccab9b-m98w6"]
Jul 23 18:44:52.761201 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:52.761165    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfcb847f-b66d-4a4c-abf1-60848315d99a" path="/var/lib/kubelet/pods/cfcb847f-b66d-4a4c-abf1-60848315d99a/volumes"
Jul 23 18:44:53.992970 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:53.992937    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"]
Jul 23 18:44:53.993380 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:53.993360    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfcb847f-b66d-4a4c-abf1-60848315d99a" containerName="revoke-keys"
Jul 23 18:44:53.993380 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:53.993371    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfcb847f-b66d-4a4c-abf1-60848315d99a" containerName="revoke-keys"
Jul 23 18:44:53.993446 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:53.993436    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfcb847f-b66d-4a4c-abf1-60848315d99a" containerName="revoke-keys"
Jul 23 18:44:53.996561 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:53.996545    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:53.999388 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:53.999366    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d24ea493-serving-cert\""
Jul 23 18:44:53.999388 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:53.999378    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 18:44:54.000333 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.000314    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:44:54.000434 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.000370    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvms6\""
Jul 23 18:44:54.009711 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.009688    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"]
Jul 23 18:44:54.089405 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.089372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjddv\" (UniqueName: \"kubernetes.io/projected/641c4631-6223-4307-b247-0a77e5e7f8e6-kube-api-access-jjddv\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.089584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.089419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c4631-6223-4307-b247-0a77e5e7f8e6-openshift-service-ca\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.089584 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.089448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641c4631-6223-4307-b247-0a77e5e7f8e6-maas-api-tls\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.189907 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.189873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjddv\" (UniqueName: \"kubernetes.io/projected/641c4631-6223-4307-b247-0a77e5e7f8e6-kube-api-access-jjddv\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.190061 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.189911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c4631-6223-4307-b247-0a77e5e7f8e6-openshift-service-ca\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.190061 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.189938    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641c4631-6223-4307-b247-0a77e5e7f8e6-maas-api-tls\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.190061 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:44:54.190049    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-serving-cert: secret "maas-api-e2e-ait-d24ea493-serving-cert" not found
Jul 23 18:44:54.190165 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:44:54.190116    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641c4631-6223-4307-b247-0a77e5e7f8e6-maas-api-tls podName:641c4631-6223-4307-b247-0a77e5e7f8e6 nodeName:}" failed. No retries permitted until 2026-07-23 18:44:54.690095276 +0000 UTC m=+1690.537785798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/641c4631-6223-4307-b247-0a77e5e7f8e6-maas-api-tls") pod "maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl" (UID: "641c4631-6223-4307-b247-0a77e5e7f8e6") : secret "maas-api-e2e-ait-d24ea493-serving-cert" not found
Jul 23 18:44:54.190581 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.190560    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c4631-6223-4307-b247-0a77e5e7f8e6-openshift-service-ca\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.199227 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.199197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjddv\" (UniqueName: \"kubernetes.io/projected/641c4631-6223-4307-b247-0a77e5e7f8e6-kube-api-access-jjddv\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.694591 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.694551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641c4631-6223-4307-b247-0a77e5e7f8e6-maas-api-tls\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.696889 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.696862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641c4631-6223-4307-b247-0a77e5e7f8e6-maas-api-tls\") pod \"maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:54.906786 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:54.906750    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:44:55.038266 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:55.038242    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"]
Jul 23 18:44:55.039978 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:44:55.039951    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641c4631_6223_4307_b247_0a77e5e7f8e6.slice/crio-cc7c408f27a5997d897d4ee4aa236ea503460d5e85c349361680df3803c2e535 WatchSource:0}: Error finding container cc7c408f27a5997d897d4ee4aa236ea503460d5e85c349361680df3803c2e535: Status 404 returned error can't find the container with id cc7c408f27a5997d897d4ee4aa236ea503460d5e85c349361680df3803c2e535
Jul 23 18:44:55.806081 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:55.806049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl" event={"ID":"641c4631-6223-4307-b247-0a77e5e7f8e6","Type":"ContainerStarted","Data":"cc7c408f27a5997d897d4ee4aa236ea503460d5e85c349361680df3803c2e535"}
Jul 23 18:44:57.819044 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:57.819005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl" event={"ID":"641c4631-6223-4307-b247-0a77e5e7f8e6","Type":"ContainerStarted","Data":"d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb"}
Jul 23 18:44:57.819421 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:44:57.819061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:45:00.137407 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.137267    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl" podStartSLOduration=4.534671056 podStartE2EDuration="7.137249627s" totalImagesPullingTime="2.602578571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:44:57.840719243 +0000 UTC m=+1693.688409770" watchObservedRunningTime="2026-07-23 18:45:00.137249627 +0000 UTC m=+1695.984940167"
Jul 23 18:45:00.139438 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.139407    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"]
Jul 23 18:45:00.142990 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.142972    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5"]
Jul 23 18:45:00.143122 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.143102    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"
Jul 23 18:45:00.146104 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.146085    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5"
Jul 23 18:45:00.147661 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.147642    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:45:00.154976 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.154948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"
Jul 23 18:45:00.155390 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.155371    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5"]
Jul 23 18:45:00.158809 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.158788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"]
Jul 23 18:45:00.163624 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.163605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5"
Jul 23 18:45:00.307527 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.307495    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"]
Jul 23 18:45:00.310502 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:45:00.310477    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27dbe587_9171_4668_99e3_246d50d7d110.slice/crio-7640116e53dc93fb367888bbff9c69091b40072610a9b492773767a7b7c856db WatchSource:0}: Error finding container 7640116e53dc93fb367888bbff9c69091b40072610a9b492773767a7b7c856db: Status 404 returned error can't find the container with id 7640116e53dc93fb367888bbff9c69091b40072610a9b492773767a7b7c856db
Jul 23 18:45:00.327197 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.327173    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5"]
Jul 23 18:45:00.329437 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:45:00.329411    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7d584f_e615_47d4_bfe5_d532c55b7068.slice/crio-c856475c99ea228a2b6621549b2053d782b3984f84a973fc65ff1f789d35eab8 WatchSource:0}: Error finding container c856475c99ea228a2b6621549b2053d782b3984f84a973fc65ff1f789d35eab8: Status 404 returned error can't find the container with id c856475c99ea228a2b6621549b2053d782b3984f84a973fc65ff1f789d35eab8
Jul 23 18:45:00.789422 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.789384    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:45:00.835687 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.835647    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv" event={"ID":"27dbe587-9171-4668-99e3-246d50d7d110","Type":"ContainerStarted","Data":"3266aa459eb8d9bd061a22ddd242a910fab1947a145531a1080e1a4f65ca88cf"}
Jul 23 18:45:00.835880 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.835715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv" event={"ID":"27dbe587-9171-4668-99e3-246d50d7d110","Type":"ContainerStarted","Data":"7640116e53dc93fb367888bbff9c69091b40072610a9b492773767a7b7c856db"}
Jul 23 18:45:00.837179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.837151    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5" event={"ID":"9f7d584f-e615-47d4-bfe5-d532c55b7068","Type":"ContainerStarted","Data":"48320a856fef8dcc492c05e8f1f04777a52c64143091b7b2a0874a657deeb200"}
Jul 23 18:45:00.837322 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.837189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5" event={"ID":"9f7d584f-e615-47d4-bfe5-d532c55b7068","Type":"ContainerStarted","Data":"c856475c99ea228a2b6621549b2053d782b3984f84a973fc65ff1f789d35eab8"}
Jul 23 18:45:00.850416 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.850369    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv" podStartSLOduration=0.850352834 podStartE2EDuration="850.352834ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:45:00.849496566 +0000 UTC m=+1696.697187093" watchObservedRunningTime="2026-07-23 18:45:00.850352834 +0000 UTC m=+1696.698043360"
Jul 23 18:45:00.866330 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:00.866271    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5" podStartSLOduration=0.866259015 podStartE2EDuration="866.259015ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:45:00.864994093 +0000 UTC m=+1696.712684633" watchObservedRunningTime="2026-07-23 18:45:00.866259015 +0000 UTC m=+1696.713949542"
Jul 23 18:45:01.842772 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:01.842733    2561 generic.go:356] "Generic (PLEG): container finished" podID="9f7d584f-e615-47d4-bfe5-d532c55b7068" containerID="48320a856fef8dcc492c05e8f1f04777a52c64143091b7b2a0874a657deeb200" exitCode=0
Jul 23 18:45:01.843241 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:01.842812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5" event={"ID":"9f7d584f-e615-47d4-bfe5-d532c55b7068","Type":"ContainerDied","Data":"48320a856fef8dcc492c05e8f1f04777a52c64143091b7b2a0874a657deeb200"}
Jul 23 18:45:01.844362 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:01.844340    2561 generic.go:356] "Generic (PLEG): container finished" podID="27dbe587-9171-4668-99e3-246d50d7d110" containerID="3266aa459eb8d9bd061a22ddd242a910fab1947a145531a1080e1a4f65ca88cf" exitCode=7
Jul 23 18:45:01.844497 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:01.844421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv" event={"ID":"27dbe587-9171-4668-99e3-246d50d7d110","Type":"ContainerDied","Data":"3266aa459eb8d9bd061a22ddd242a910fab1947a145531a1080e1a4f65ca88cf"}
Jul 23 18:45:01.844578 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:01.844561    2561 scope.go:117] "RemoveContainer" containerID="3266aa459eb8d9bd061a22ddd242a910fab1947a145531a1080e1a4f65ca88cf"
Jul 23 18:45:02.851246 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:02.851163    2561 generic.go:356] "Generic (PLEG): container finished" podID="27dbe587-9171-4668-99e3-246d50d7d110" containerID="f028a9a0cdf539f9509d2d8130a77b2029750d335fdfd884995245639c774242" exitCode=7
Jul 23 18:45:02.851246 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:02.851231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv" event={"ID":"27dbe587-9171-4668-99e3-246d50d7d110","Type":"ContainerDied","Data":"f028a9a0cdf539f9509d2d8130a77b2029750d335fdfd884995245639c774242"}
Jul 23 18:45:02.851742 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:02.851296    2561 scope.go:117] "RemoveContainer" containerID="3266aa459eb8d9bd061a22ddd242a910fab1947a145531a1080e1a4f65ca88cf"
Jul 23 18:45:02.851742 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:02.851432    2561 scope.go:117] "RemoveContainer" containerID="f028a9a0cdf539f9509d2d8130a77b2029750d335fdfd884995245639c774242"
Jul 23 18:45:02.851742 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:45:02.851717    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv_odh-ai-gateway-infra(27dbe587-9171-4668-99e3-246d50d7d110)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv" podUID="27dbe587-9171-4668-99e3-246d50d7d110"
Jul 23 18:45:03.004138 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:03.004115    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5"
Jul 23 18:45:03.831186 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:03.831155    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:45:03.857740 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:03.857711    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5"
Jul 23 18:45:03.858118 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:03.857748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747205-xt8k5" event={"ID":"9f7d584f-e615-47d4-bfe5-d532c55b7068","Type":"ContainerDied","Data":"c856475c99ea228a2b6621549b2053d782b3984f84a973fc65ff1f789d35eab8"}
Jul 23 18:45:03.858118 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:03.857798    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c856475c99ea228a2b6621549b2053d782b3984f84a973fc65ff1f789d35eab8"
Jul 23 18:45:03.859461 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:03.859443    2561 scope.go:117] "RemoveContainer" containerID="f028a9a0cdf539f9509d2d8130a77b2029750d335fdfd884995245639c774242"
Jul 23 18:45:03.859687 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:45:03.859668    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv_odh-ai-gateway-infra(27dbe587-9171-4668-99e3-246d50d7d110)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv" podUID="27dbe587-9171-4668-99e3-246d50d7d110"
Jul 23 18:45:17.755634 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:17.755599    2561 scope.go:117] "RemoveContainer" containerID="f028a9a0cdf539f9509d2d8130a77b2029750d335fdfd884995245639c774242"
Jul 23 18:45:18.936553 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:18.936477    2561 generic.go:356] "Generic (PLEG): container finished" podID="27dbe587-9171-4668-99e3-246d50d7d110" containerID="304365a05193efd45a4063304a4a5f5626b9f7f17834addce0a65ff8fa974410" exitCode=0
Jul 23 18:45:18.936951 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:18.936565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv" event={"ID":"27dbe587-9171-4668-99e3-246d50d7d110","Type":"ContainerDied","Data":"304365a05193efd45a4063304a4a5f5626b9f7f17834addce0a65ff8fa974410"}
Jul 23 18:45:18.936951 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:18.936647    2561 scope.go:117] "RemoveContainer" containerID="f028a9a0cdf539f9509d2d8130a77b2029750d335fdfd884995245639c774242"
Jul 23 18:45:19.613265 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.613231    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"]
Jul 23 18:45:19.613658 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.613644    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7d584f-e615-47d4-bfe5-d532c55b7068" containerName="cleanup"
Jul 23 18:45:19.613704 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.613659    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7d584f-e615-47d4-bfe5-d532c55b7068" containerName="cleanup"
Jul 23 18:45:19.613739 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.613732    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7d584f-e615-47d4-bfe5-d532c55b7068" containerName="cleanup"
Jul 23 18:45:19.618424 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.618405    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"
Jul 23 18:45:19.625069 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.625035    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"]
Jul 23 18:45:19.725724 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.725687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47e54ce-bb48-45b9-bd1f-231371def7da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d24ea493-npf4s\" (UID: \"c47e54ce-bb48-45b9-bd1f-231371def7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"
Jul 23 18:45:19.826732 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.826697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47e54ce-bb48-45b9-bd1f-231371def7da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d24ea493-npf4s\" (UID: \"c47e54ce-bb48-45b9-bd1f-231371def7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"
Jul 23 18:45:19.827257 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.827237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47e54ce-bb48-45b9-bd1f-231371def7da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d24ea493-npf4s\" (UID: \"c47e54ce-bb48-45b9-bd1f-231371def7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"
Jul 23 18:45:19.931935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.931861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"
Jul 23 18:45:19.964875 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:19.964835    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"]
Jul 23 18:45:20.082780 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:20.082745    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"]
Jul 23 18:45:20.085136 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:45:20.085106    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47e54ce_bb48_45b9_bd1f_231371def7da.slice/crio-7d975d1338ca2ea00a1becfc3ce8c15aae7da430d34515f80d79c8cf4d78ff73 WatchSource:0}: Error finding container 7d975d1338ca2ea00a1becfc3ce8c15aae7da430d34515f80d79c8cf4d78ff73: Status 404 returned error can't find the container with id 7d975d1338ca2ea00a1becfc3ce8c15aae7da430d34515f80d79c8cf4d78ff73
Jul 23 18:45:20.098029 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:20.098011    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"
Jul 23 18:45:20.949413 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:20.949381    2561 generic.go:356] "Generic (PLEG): container finished" podID="c47e54ce-bb48-45b9-bd1f-231371def7da" containerID="d102b9893ae8e4dfda9f23276ee58d185a35693514332993afba24a85a20a944" exitCode=0
Jul 23 18:45:20.949604 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:20.949479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s" event={"ID":"c47e54ce-bb48-45b9-bd1f-231371def7da","Type":"ContainerDied","Data":"d102b9893ae8e4dfda9f23276ee58d185a35693514332993afba24a85a20a944"}
Jul 23 18:45:20.949604 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:20.949533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s" event={"ID":"c47e54ce-bb48-45b9-bd1f-231371def7da","Type":"ContainerStarted","Data":"7d975d1338ca2ea00a1becfc3ce8c15aae7da430d34515f80d79c8cf4d78ff73"}
Jul 23 18:45:20.950732 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:20.950708    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv" event={"ID":"27dbe587-9171-4668-99e3-246d50d7d110","Type":"ContainerDied","Data":"7640116e53dc93fb367888bbff9c69091b40072610a9b492773767a7b7c856db"}
Jul 23 18:45:20.950732 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:20.950734    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"
Jul 23 18:45:20.950873 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:20.950754    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7640116e53dc93fb367888bbff9c69091b40072610a9b492773767a7b7c856db"
Jul 23 18:45:20.999780 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:20.999704    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"]
Jul 23 18:45:21.004944 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:21.004912    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-d24ea493-29747205-bpdnv"]
Jul 23 18:45:22.088539 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:22.088513    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"
Jul 23 18:45:22.147213 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:22.147180    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47e54ce-bb48-45b9-bd1f-231371def7da-maas-api-service-ca\") pod \"c47e54ce-bb48-45b9-bd1f-231371def7da\" (UID: \"c47e54ce-bb48-45b9-bd1f-231371def7da\") "
Jul 23 18:45:22.147570 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:22.147547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47e54ce-bb48-45b9-bd1f-231371def7da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c47e54ce-bb48-45b9-bd1f-231371def7da" (UID: "c47e54ce-bb48-45b9-bd1f-231371def7da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:45:22.248496 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:22.248422    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47e54ce-bb48-45b9-bd1f-231371def7da-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:45:22.761045 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:22.761008    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27dbe587-9171-4668-99e3-246d50d7d110" path="/var/lib/kubelet/pods/27dbe587-9171-4668-99e3-246d50d7d110/volumes"
Jul 23 18:45:22.962484 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:22.962455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s" event={"ID":"c47e54ce-bb48-45b9-bd1f-231371def7da","Type":"ContainerDied","Data":"7d975d1338ca2ea00a1becfc3ce8c15aae7da430d34515f80d79c8cf4d78ff73"}
Jul 23 18:45:22.962655 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:22.962493    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"
Jul 23 18:45:22.962655 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:22.962508    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d975d1338ca2ea00a1becfc3ce8c15aae7da430d34515f80d79c8cf4d78ff73"
Jul 23 18:45:30.226793 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.226761    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"]
Jul 23 18:45:30.227270 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.226951    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl" podUID="641c4631-6223-4307-b247-0a77e5e7f8e6" containerName="maas-api" containerID="cri-o://d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb" gracePeriod=30
Jul 23 18:45:30.470867 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.470840    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:45:30.528868 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.528790    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641c4631-6223-4307-b247-0a77e5e7f8e6-maas-api-tls\") pod \"641c4631-6223-4307-b247-0a77e5e7f8e6\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") "
Jul 23 18:45:30.528868 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.528865    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjddv\" (UniqueName: \"kubernetes.io/projected/641c4631-6223-4307-b247-0a77e5e7f8e6-kube-api-access-jjddv\") pod \"641c4631-6223-4307-b247-0a77e5e7f8e6\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") "
Jul 23 18:45:30.529074 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.528893    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c4631-6223-4307-b247-0a77e5e7f8e6-openshift-service-ca\") pod \"641c4631-6223-4307-b247-0a77e5e7f8e6\" (UID: \"641c4631-6223-4307-b247-0a77e5e7f8e6\") "
Jul 23 18:45:30.529322 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.529271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641c4631-6223-4307-b247-0a77e5e7f8e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "641c4631-6223-4307-b247-0a77e5e7f8e6" (UID: "641c4631-6223-4307-b247-0a77e5e7f8e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:45:30.531077 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.531053    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641c4631-6223-4307-b247-0a77e5e7f8e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "641c4631-6223-4307-b247-0a77e5e7f8e6" (UID: "641c4631-6223-4307-b247-0a77e5e7f8e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:45:30.531170 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.531057    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641c4631-6223-4307-b247-0a77e5e7f8e6-kube-api-access-jjddv" (OuterVolumeSpecName: "kube-api-access-jjddv") pod "641c4631-6223-4307-b247-0a77e5e7f8e6" (UID: "641c4631-6223-4307-b247-0a77e5e7f8e6"). InnerVolumeSpecName "kube-api-access-jjddv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:45:30.630406 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.630378    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjddv\" (UniqueName: \"kubernetes.io/projected/641c4631-6223-4307-b247-0a77e5e7f8e6-kube-api-access-jjddv\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:45:30.630406 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.630401    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c4631-6223-4307-b247-0a77e5e7f8e6-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:45:30.630406 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:30.630411    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641c4631-6223-4307-b247-0a77e5e7f8e6-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:45:31.002991 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:31.002952    2561 generic.go:356] "Generic (PLEG): container finished" podID="641c4631-6223-4307-b247-0a77e5e7f8e6" containerID="d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb" exitCode=0
Jul 23 18:45:31.003191 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:31.002996    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"
Jul 23 18:45:31.003191 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:31.003034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl" event={"ID":"641c4631-6223-4307-b247-0a77e5e7f8e6","Type":"ContainerDied","Data":"d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb"}
Jul 23 18:45:31.003191 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:31.003088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl" event={"ID":"641c4631-6223-4307-b247-0a77e5e7f8e6","Type":"ContainerDied","Data":"cc7c408f27a5997d897d4ee4aa236ea503460d5e85c349361680df3803c2e535"}
Jul 23 18:45:31.003191 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:31.003116    2561 scope.go:117] "RemoveContainer" containerID="d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb"
Jul 23 18:45:31.011780 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:31.011762    2561 scope.go:117] "RemoveContainer" containerID="d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb"
Jul 23 18:45:31.012045 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:45:31.012024    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb\": container with ID starting with d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb not found: ID does not exist" containerID="d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb"
Jul 23 18:45:31.012095 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:31.012053    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb"} err="failed to get container status \"d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb\": rpc error: code = NotFound desc = could not find container \"d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb\": container with ID starting with d289a9e3c02186fd1b830f9279231ef450cb398dcfbf2d3b1a0394e469b945fb not found: ID does not exist"
Jul 23 18:45:31.022500 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:31.022467    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"]
Jul 23 18:45:31.029876 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:31.029854    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d24ea493-8664c7b757-kf7bl"]
Jul 23 18:45:32.760534 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:32.760498    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="641c4631-6223-4307-b247-0a77e5e7f8e6" path="/var/lib/kubelet/pods/641c4631-6223-4307-b247-0a77e5e7f8e6/volumes"
Jul 23 18:45:32.996061 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:32.996031    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:45:34.734998 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:34.734958    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"]
Jul 23 18:45:34.737883 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:34.737846    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d24ea493-npf4s"]
Jul 23 18:45:34.762119 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:34.762088    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c47e54ce-bb48-45b9-bd1f-231371def7da" path="/var/lib/kubelet/pods/c47e54ce-bb48-45b9-bd1f-231371def7da/volumes"
Jul 23 18:45:39.200491 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200458    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"]
Jul 23 18:45:39.200866 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200850    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27dbe587-9171-4668-99e3-246d50d7d110" containerName="cleanup"
Jul 23 18:45:39.200866 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200867    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27dbe587-9171-4668-99e3-246d50d7d110" containerName="cleanup"
Jul 23 18:45:39.200935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200881    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641c4631-6223-4307-b247-0a77e5e7f8e6" containerName="maas-api"
Jul 23 18:45:39.200935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200886    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="641c4631-6223-4307-b247-0a77e5e7f8e6" containerName="maas-api"
Jul 23 18:45:39.200935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200893    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27dbe587-9171-4668-99e3-246d50d7d110" containerName="cleanup"
Jul 23 18:45:39.200935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200899    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27dbe587-9171-4668-99e3-246d50d7d110" containerName="cleanup"
Jul 23 18:45:39.200935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200906    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47e54ce-bb48-45b9-bd1f-231371def7da" containerName="revoke-keys"
Jul 23 18:45:39.200935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200915    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47e54ce-bb48-45b9-bd1f-231371def7da" containerName="revoke-keys"
Jul 23 18:45:39.200935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200932    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27dbe587-9171-4668-99e3-246d50d7d110" containerName="cleanup"
Jul 23 18:45:39.200935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200937    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27dbe587-9171-4668-99e3-246d50d7d110" containerName="cleanup"
Jul 23 18:45:39.201169 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.200995    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27dbe587-9171-4668-99e3-246d50d7d110" containerName="cleanup"
Jul 23 18:45:39.201169 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.201004    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47e54ce-bb48-45b9-bd1f-231371def7da" containerName="revoke-keys"
Jul 23 18:45:39.201169 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.201012    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27dbe587-9171-4668-99e3-246d50d7d110" containerName="cleanup"
Jul 23 18:45:39.201169 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.201019    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27dbe587-9171-4668-99e3-246d50d7d110" containerName="cleanup"
Jul 23 18:45:39.201169 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.201027    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="641c4631-6223-4307-b247-0a77e5e7f8e6" containerName="maas-api"
Jul 23 18:45:39.205345 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.205326    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.209070 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.209042    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0a3d054e-serving-cert\""
Jul 23 18:45:39.209388 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.209369    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvms6\""
Jul 23 18:45:39.210014 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.210000    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 18:45:39.211929 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.211803    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:45:39.217743 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.217722    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"]
Jul 23 18:45:39.309967 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.309933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38cd37-e46d-495f-967d-0524cc4edf61-maas-api-tls\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.310132 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.310016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzm2\" (UniqueName: \"kubernetes.io/projected/cf38cd37-e46d-495f-967d-0524cc4edf61-kube-api-access-6nzm2\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.310132 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.310064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38cd37-e46d-495f-967d-0524cc4edf61-openshift-service-ca\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.411439 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.411407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzm2\" (UniqueName: \"kubernetes.io/projected/cf38cd37-e46d-495f-967d-0524cc4edf61-kube-api-access-6nzm2\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.411599 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.411450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38cd37-e46d-495f-967d-0524cc4edf61-openshift-service-ca\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.411599 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.411479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38cd37-e46d-495f-967d-0524cc4edf61-maas-api-tls\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.411672 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:45:39.411655    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-serving-cert: secret "maas-api-e2e-derive-0a3d054e-serving-cert" not found
Jul 23 18:45:39.411725 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:45:39.411714    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf38cd37-e46d-495f-967d-0524cc4edf61-maas-api-tls podName:cf38cd37-e46d-495f-967d-0524cc4edf61 nodeName:}" failed. No retries permitted until 2026-07-23 18:45:39.9116987 +0000 UTC m=+1735.759389205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf38cd37-e46d-495f-967d-0524cc4edf61-maas-api-tls") pod "maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh" (UID: "cf38cd37-e46d-495f-967d-0524cc4edf61") : secret "maas-api-e2e-derive-0a3d054e-serving-cert" not found
Jul 23 18:45:39.412129 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.412111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38cd37-e46d-495f-967d-0524cc4edf61-openshift-service-ca\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.423605 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.423575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzm2\" (UniqueName: \"kubernetes.io/projected/cf38cd37-e46d-495f-967d-0524cc4edf61-kube-api-access-6nzm2\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.917380 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.917350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38cd37-e46d-495f-967d-0524cc4edf61-maas-api-tls\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:39.919736 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:39.919705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38cd37-e46d-495f-967d-0524cc4edf61-maas-api-tls\") pod \"maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:40.115425 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:40.115382    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:40.259882 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:40.259718    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"]
Jul 23 18:45:40.262823 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:45:40.262791    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf38cd37_e46d_495f_967d_0524cc4edf61.slice/crio-a5e1e447ca833341ec39a041270943c502eb4bd1fff63e7c7f6dedc1b9efba79 WatchSource:0}: Error finding container a5e1e447ca833341ec39a041270943c502eb4bd1fff63e7c7f6dedc1b9efba79: Status 404 returned error can't find the container with id a5e1e447ca833341ec39a041270943c502eb4bd1fff63e7c7f6dedc1b9efba79
Jul 23 18:45:41.054896 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:41.054862    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh" event={"ID":"cf38cd37-e46d-495f-967d-0524cc4edf61","Type":"ContainerStarted","Data":"28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a"}
Jul 23 18:45:41.055057 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:41.054938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh" event={"ID":"cf38cd37-e46d-495f-967d-0524cc4edf61","Type":"ContainerStarted","Data":"a5e1e447ca833341ec39a041270943c502eb4bd1fff63e7c7f6dedc1b9efba79"}
Jul 23 18:45:41.055057 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:41.054986    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:45:41.072746 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:41.072698    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh" podStartSLOduration=1.650482318 podStartE2EDuration="2.072684016s" totalImagesPullingTime="422.201698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:45:41.070841983 +0000 UTC m=+1736.918532511" watchObservedRunningTime="2026-07-23 18:45:41.072684016 +0000 UTC m=+1736.920374543"
Jul 23 18:45:44.509727 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:44.509693    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:45:47.064848 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:45:47.064815    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:46:03.992527 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:03.992494    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"]
Jul 23 18:46:03.996211 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:03.996194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"
Jul 23 18:46:04.001174 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:04.001148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:46:04.013696 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:04.013671    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"]
Jul 23 18:46:04.037219 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:04.037190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca730837-5915-4be7-92ca-3d4678bb3488-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5\" (UID: \"ca730837-5915-4be7-92ca-3d4678bb3488\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"
Jul 23 18:46:04.138714 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:04.138682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca730837-5915-4be7-92ca-3d4678bb3488-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5\" (UID: \"ca730837-5915-4be7-92ca-3d4678bb3488\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"
Jul 23 18:46:04.139264 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:04.139245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca730837-5915-4be7-92ca-3d4678bb3488-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5\" (UID: \"ca730837-5915-4be7-92ca-3d4678bb3488\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"
Jul 23 18:46:04.306267 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:04.306174    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"
Jul 23 18:46:04.436084 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:04.436058    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"]
Jul 23 18:46:04.438848 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:46:04.438817    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca730837_5915_4be7_92ca_3d4678bb3488.slice/crio-fe607e45d294e0c2868ef11cc98d92a8a388868ba1e2cac67d441b987a2e2623 WatchSource:0}: Error finding container fe607e45d294e0c2868ef11cc98d92a8a388868ba1e2cac67d441b987a2e2623: Status 404 returned error can't find the container with id fe607e45d294e0c2868ef11cc98d92a8a388868ba1e2cac67d441b987a2e2623
Jul 23 18:46:05.172824 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:05.172797    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca730837-5915-4be7-92ca-3d4678bb3488" containerID="afb25ea62eb10212bfc4e1d7f3ce39bf889b298ead0f0c52c5cabde6859030d9" exitCode=0
Jul 23 18:46:05.173155 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:05.172871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5" event={"ID":"ca730837-5915-4be7-92ca-3d4678bb3488","Type":"ContainerDied","Data":"afb25ea62eb10212bfc4e1d7f3ce39bf889b298ead0f0c52c5cabde6859030d9"}
Jul 23 18:46:05.173155 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:05.172948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5" event={"ID":"ca730837-5915-4be7-92ca-3d4678bb3488","Type":"ContainerStarted","Data":"fe607e45d294e0c2868ef11cc98d92a8a388868ba1e2cac67d441b987a2e2623"}
Jul 23 18:46:06.304698 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:06.304674    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"
Jul 23 18:46:06.360232 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:06.360196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca730837-5915-4be7-92ca-3d4678bb3488-maas-api-service-ca\") pod \"ca730837-5915-4be7-92ca-3d4678bb3488\" (UID: \"ca730837-5915-4be7-92ca-3d4678bb3488\") "
Jul 23 18:46:06.360576 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:06.360554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca730837-5915-4be7-92ca-3d4678bb3488-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca730837-5915-4be7-92ca-3d4678bb3488" (UID: "ca730837-5915-4be7-92ca-3d4678bb3488"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:46:06.461698 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:06.461671    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca730837-5915-4be7-92ca-3d4678bb3488-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:46:07.184502 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:07.184429    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"
Jul 23 18:46:07.184502 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:07.184483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5" event={"ID":"ca730837-5915-4be7-92ca-3d4678bb3488","Type":"ContainerDied","Data":"fe607e45d294e0c2868ef11cc98d92a8a388868ba1e2cac67d441b987a2e2623"}
Jul 23 18:46:07.184675 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:07.184524    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe607e45d294e0c2868ef11cc98d92a8a388868ba1e2cac67d441b987a2e2623"
Jul 23 18:46:14.565269 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:14.565236    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"]
Jul 23 18:46:14.565794 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:14.565513    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh" podUID="cf38cd37-e46d-495f-967d-0524cc4edf61" containerName="maas-api" containerID="cri-o://28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a" gracePeriod=30
Jul 23 18:46:14.809817 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:14.809793    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:46:14.933015 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:14.932942    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38cd37-e46d-495f-967d-0524cc4edf61-maas-api-tls\") pod \"cf38cd37-e46d-495f-967d-0524cc4edf61\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") "
Jul 23 18:46:14.933154 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:14.933021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38cd37-e46d-495f-967d-0524cc4edf61-openshift-service-ca\") pod \"cf38cd37-e46d-495f-967d-0524cc4edf61\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") "
Jul 23 18:46:14.933154 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:14.933060    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nzm2\" (UniqueName: \"kubernetes.io/projected/cf38cd37-e46d-495f-967d-0524cc4edf61-kube-api-access-6nzm2\") pod \"cf38cd37-e46d-495f-967d-0524cc4edf61\" (UID: \"cf38cd37-e46d-495f-967d-0524cc4edf61\") "
Jul 23 18:46:14.933431 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:14.933411    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf38cd37-e46d-495f-967d-0524cc4edf61-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf38cd37-e46d-495f-967d-0524cc4edf61" (UID: "cf38cd37-e46d-495f-967d-0524cc4edf61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:46:14.934996 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:14.934976    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf38cd37-e46d-495f-967d-0524cc4edf61-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf38cd37-e46d-495f-967d-0524cc4edf61" (UID: "cf38cd37-e46d-495f-967d-0524cc4edf61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:46:14.935089 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:14.935026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf38cd37-e46d-495f-967d-0524cc4edf61-kube-api-access-6nzm2" (OuterVolumeSpecName: "kube-api-access-6nzm2") pod "cf38cd37-e46d-495f-967d-0524cc4edf61" (UID: "cf38cd37-e46d-495f-967d-0524cc4edf61"). InnerVolumeSpecName "kube-api-access-6nzm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:46:15.034214 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.034170    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38cd37-e46d-495f-967d-0524cc4edf61-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:46:15.034214 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.034212    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38cd37-e46d-495f-967d-0524cc4edf61-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:46:15.034214 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.034223    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nzm2\" (UniqueName: \"kubernetes.io/projected/cf38cd37-e46d-495f-967d-0524cc4edf61-kube-api-access-6nzm2\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:46:15.226452 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.226421    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf38cd37-e46d-495f-967d-0524cc4edf61" containerID="28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a" exitCode=0
Jul 23 18:46:15.226601 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.226463    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"
Jul 23 18:46:15.226601 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.226496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh" event={"ID":"cf38cd37-e46d-495f-967d-0524cc4edf61","Type":"ContainerDied","Data":"28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a"}
Jul 23 18:46:15.226601 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.226536    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh" event={"ID":"cf38cd37-e46d-495f-967d-0524cc4edf61","Type":"ContainerDied","Data":"a5e1e447ca833341ec39a041270943c502eb4bd1fff63e7c7f6dedc1b9efba79"}
Jul 23 18:46:15.226601 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.226558    2561 scope.go:117] "RemoveContainer" containerID="28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a"
Jul 23 18:46:15.235325 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.235279    2561 scope.go:117] "RemoveContainer" containerID="28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a"
Jul 23 18:46:15.235569 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:46:15.235549    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a\": container with ID starting with 28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a not found: ID does not exist" containerID="28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a"
Jul 23 18:46:15.235619 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.235575    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a"} err="failed to get container status \"28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a\": rpc error: code = NotFound desc = could not find container \"28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a\": container with ID starting with 28c9414c4b3335b186fe82deabc92cafb5e64bfbf2324ec0c7c093734e65539a not found: ID does not exist"
Jul 23 18:46:15.247650 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.247627    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"]
Jul 23 18:46:15.251852 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:15.251831    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3d054e-7f4dfb9bd6-6mfsh"]
Jul 23 18:46:16.760924 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:16.760887    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf38cd37-e46d-495f-967d-0524cc4edf61" path="/var/lib/kubelet/pods/cf38cd37-e46d-495f-967d-0524cc4edf61/volumes"
Jul 23 18:46:17.489809 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:17.489772    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:46:19.064324 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:19.064272    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"]
Jul 23 18:46:19.067883 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:19.067858    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3d054e-88bc5"]
Jul 23 18:46:20.760818 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:20.760781    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca730837-5915-4be7-92ca-3d4678bb3488" path="/var/lib/kubelet/pods/ca730837-5915-4be7-92ca-3d4678bb3488/volumes"
Jul 23 18:46:44.725087 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:44.725059    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:46:44.731473 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:46:44.731450    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:47:50.294179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:47:50.294145    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:48:02.603512 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:02.603475    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:48:06.505420 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.505338    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"]
Jul 23 18:48:06.505778 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.505722    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca730837-5915-4be7-92ca-3d4678bb3488" containerName="revoke-keys"
Jul 23 18:48:06.505778 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.505735    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca730837-5915-4be7-92ca-3d4678bb3488" containerName="revoke-keys"
Jul 23 18:48:06.505778 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.505751    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf38cd37-e46d-495f-967d-0524cc4edf61" containerName="maas-api"
Jul 23 18:48:06.505778 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.505756    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf38cd37-e46d-495f-967d-0524cc4edf61" containerName="maas-api"
Jul 23 18:48:06.505915 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.505817    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf38cd37-e46d-495f-967d-0524cc4edf61" containerName="maas-api"
Jul 23 18:48:06.505915 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.505827    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca730837-5915-4be7-92ca-3d4678bb3488" containerName="revoke-keys"
Jul 23 18:48:06.508892 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.508873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:06.512394 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.512377    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4ce7c7da-serving-cert\""
Jul 23 18:48:06.512636 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.512618    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:48:06.513416 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.513402    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 18:48:06.513468 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.513414    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvms6\""
Jul 23 18:48:06.522675 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.522649    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"]
Jul 23 18:48:06.615164 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.615130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfaa5798-16de-4e0a-9c9e-200d8099a099-maas-api-tls\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:06.615164 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.615165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sz8g\" (UniqueName: \"kubernetes.io/projected/cfaa5798-16de-4e0a-9c9e-200d8099a099-kube-api-access-6sz8g\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:06.615373 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.615193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa5798-16de-4e0a-9c9e-200d8099a099-openshift-service-ca\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:06.718066 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.716519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa5798-16de-4e0a-9c9e-200d8099a099-openshift-service-ca\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:06.718066 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.716719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfaa5798-16de-4e0a-9c9e-200d8099a099-maas-api-tls\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:06.718066 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.716758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sz8g\" (UniqueName: \"kubernetes.io/projected/cfaa5798-16de-4e0a-9c9e-200d8099a099-kube-api-access-6sz8g\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:06.718066 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:48:06.717703    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-serving-cert: secret "maas-api-e2e-mt-4ce7c7da-serving-cert" not found
Jul 23 18:48:06.718066 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:48:06.717784    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfaa5798-16de-4e0a-9c9e-200d8099a099-maas-api-tls podName:cfaa5798-16de-4e0a-9c9e-200d8099a099 nodeName:}" failed. No retries permitted until 2026-07-23 18:48:07.217762169 +0000 UTC m=+1883.065452676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfaa5798-16de-4e0a-9c9e-200d8099a099-maas-api-tls") pod "maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7" (UID: "cfaa5798-16de-4e0a-9c9e-200d8099a099") : secret "maas-api-e2e-mt-4ce7c7da-serving-cert" not found
Jul 23 18:48:06.718873 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.718847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa5798-16de-4e0a-9c9e-200d8099a099-openshift-service-ca\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:06.729756 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:06.729723    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sz8g\" (UniqueName: \"kubernetes.io/projected/cfaa5798-16de-4e0a-9c9e-200d8099a099-kube-api-access-6sz8g\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:07.221914 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:07.221876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfaa5798-16de-4e0a-9c9e-200d8099a099-maas-api-tls\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:07.224376 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:07.224352    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfaa5798-16de-4e0a-9c9e-200d8099a099-maas-api-tls\") pod \"maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:07.419697 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:07.419664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:07.554155 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:07.554131    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"]
Jul 23 18:48:07.556454 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:48:07.556424    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaa5798_16de_4e0a_9c9e_200d8099a099.slice/crio-1cd0d2b6e933453a1c3970bc4b5f314c1fe3950154c1f8edf404a94df41d6fa6 WatchSource:0}: Error finding container 1cd0d2b6e933453a1c3970bc4b5f314c1fe3950154c1f8edf404a94df41d6fa6: Status 404 returned error can't find the container with id 1cd0d2b6e933453a1c3970bc4b5f314c1fe3950154c1f8edf404a94df41d6fa6
Jul 23 18:48:07.780151 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:07.780061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7" event={"ID":"cfaa5798-16de-4e0a-9c9e-200d8099a099","Type":"ContainerStarted","Data":"1cd0d2b6e933453a1c3970bc4b5f314c1fe3950154c1f8edf404a94df41d6fa6"}
Jul 23 18:48:08.786359 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:08.786320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7" event={"ID":"cfaa5798-16de-4e0a-9c9e-200d8099a099","Type":"ContainerStarted","Data":"f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c"}
Jul 23 18:48:08.786791 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:08.786390    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:08.806118 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:08.806066    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7" podStartSLOduration=2.219659468 podStartE2EDuration="2.806049754s" totalImagesPullingTime="586.390286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:48:08.804396752 +0000 UTC m=+1884.652087279" watchObservedRunningTime="2026-07-23 18:48:08.806049754 +0000 UTC m=+1884.653740281"
Jul 23 18:48:14.797488 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:14.797454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:16.783822 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:16.783794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:48:18.555631 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:18.555593    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"]
Jul 23 18:48:18.559104 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:18.559085    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"
Jul 23 18:48:18.562518 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:18.562500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:48:18.577269 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:18.577230    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"]
Jul 23 18:48:18.611632 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:18.611601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d4a32-0457-42bd-837e-05306cd484ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7\" (UID: \"c66d4a32-0457-42bd-837e-05306cd484ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"
Jul 23 18:48:18.712228 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:18.712194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d4a32-0457-42bd-837e-05306cd484ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7\" (UID: \"c66d4a32-0457-42bd-837e-05306cd484ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"
Jul 23 18:48:18.712802 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:18.712782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d4a32-0457-42bd-837e-05306cd484ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7\" (UID: \"c66d4a32-0457-42bd-837e-05306cd484ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"
Jul 23 18:48:18.869299 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:18.869198    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"
Jul 23 18:48:18.996710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:18.996683    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"]
Jul 23 18:48:18.999023 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:48:18.998988    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66d4a32_0457_42bd_837e_05306cd484ae.slice/crio-da8322c64d5ec1c329f5aa2fd3b138475bf7c7362b1aa363e51caaa373873fc2 WatchSource:0}: Error finding container da8322c64d5ec1c329f5aa2fd3b138475bf7c7362b1aa363e51caaa373873fc2: Status 404 returned error can't find the container with id da8322c64d5ec1c329f5aa2fd3b138475bf7c7362b1aa363e51caaa373873fc2
Jul 23 18:48:19.846063 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:19.846034    2561 generic.go:356] "Generic (PLEG): container finished" podID="c66d4a32-0457-42bd-837e-05306cd484ae" containerID="e44a570809b2e0b378473043e1b0efa45fefcf57acae9ee6284e116bbc4fcf23" exitCode=0
Jul 23 18:48:19.846404 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:19.846110    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7" event={"ID":"c66d4a32-0457-42bd-837e-05306cd484ae","Type":"ContainerDied","Data":"e44a570809b2e0b378473043e1b0efa45fefcf57acae9ee6284e116bbc4fcf23"}
Jul 23 18:48:19.846404 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:19.846156    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7" event={"ID":"c66d4a32-0457-42bd-837e-05306cd484ae","Type":"ContainerStarted","Data":"da8322c64d5ec1c329f5aa2fd3b138475bf7c7362b1aa363e51caaa373873fc2"}
Jul 23 18:48:20.982030 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:20.982006    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"
Jul 23 18:48:21.030088 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:21.030057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d4a32-0457-42bd-837e-05306cd484ae-maas-api-service-ca\") pod \"c66d4a32-0457-42bd-837e-05306cd484ae\" (UID: \"c66d4a32-0457-42bd-837e-05306cd484ae\") "
Jul 23 18:48:21.030460 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:21.030438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66d4a32-0457-42bd-837e-05306cd484ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c66d4a32-0457-42bd-837e-05306cd484ae" (UID: "c66d4a32-0457-42bd-837e-05306cd484ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:48:21.131429 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:21.131351    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d4a32-0457-42bd-837e-05306cd484ae-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:48:21.857426 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:21.857392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7" event={"ID":"c66d4a32-0457-42bd-837e-05306cd484ae","Type":"ContainerDied","Data":"da8322c64d5ec1c329f5aa2fd3b138475bf7c7362b1aa363e51caaa373873fc2"}
Jul 23 18:48:21.857426 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:21.857428    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"
Jul 23 18:48:21.857722 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:21.857440    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da8322c64d5ec1c329f5aa2fd3b138475bf7c7362b1aa363e51caaa373873fc2"
Jul 23 18:48:29.033627 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.033595    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"]
Jul 23 18:48:29.034056 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.033808    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7" podUID="cfaa5798-16de-4e0a-9c9e-200d8099a099" containerName="maas-api" containerID="cri-o://f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c" gracePeriod=30
Jul 23 18:48:29.290217 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.290153    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:29.400365 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.400331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfaa5798-16de-4e0a-9c9e-200d8099a099-maas-api-tls\") pod \"cfaa5798-16de-4e0a-9c9e-200d8099a099\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") "
Jul 23 18:48:29.400526 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.400390    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa5798-16de-4e0a-9c9e-200d8099a099-openshift-service-ca\") pod \"cfaa5798-16de-4e0a-9c9e-200d8099a099\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") "
Jul 23 18:48:29.400526 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.400462    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sz8g\" (UniqueName: \"kubernetes.io/projected/cfaa5798-16de-4e0a-9c9e-200d8099a099-kube-api-access-6sz8g\") pod \"cfaa5798-16de-4e0a-9c9e-200d8099a099\" (UID: \"cfaa5798-16de-4e0a-9c9e-200d8099a099\") "
Jul 23 18:48:29.400730 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.400709    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaa5798-16de-4e0a-9c9e-200d8099a099-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cfaa5798-16de-4e0a-9c9e-200d8099a099" (UID: "cfaa5798-16de-4e0a-9c9e-200d8099a099"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:48:29.402552 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.402521    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaa5798-16de-4e0a-9c9e-200d8099a099-kube-api-access-6sz8g" (OuterVolumeSpecName: "kube-api-access-6sz8g") pod "cfaa5798-16de-4e0a-9c9e-200d8099a099" (UID: "cfaa5798-16de-4e0a-9c9e-200d8099a099"). InnerVolumeSpecName "kube-api-access-6sz8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:48:29.402634 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.402613    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfaa5798-16de-4e0a-9c9e-200d8099a099-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cfaa5798-16de-4e0a-9c9e-200d8099a099" (UID: "cfaa5798-16de-4e0a-9c9e-200d8099a099"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:48:29.501689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.501644    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sz8g\" (UniqueName: \"kubernetes.io/projected/cfaa5798-16de-4e0a-9c9e-200d8099a099-kube-api-access-6sz8g\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:48:29.501689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.501685    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfaa5798-16de-4e0a-9c9e-200d8099a099-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:48:29.501689 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.501694    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa5798-16de-4e0a-9c9e-200d8099a099-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:48:29.898026 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.897987    2561 generic.go:356] "Generic (PLEG): container finished" podID="cfaa5798-16de-4e0a-9c9e-200d8099a099" containerID="f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c" exitCode=0
Jul 23 18:48:29.898221 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.898050    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"
Jul 23 18:48:29.898221 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.898076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7" event={"ID":"cfaa5798-16de-4e0a-9c9e-200d8099a099","Type":"ContainerDied","Data":"f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c"}
Jul 23 18:48:29.898221 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.898131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7" event={"ID":"cfaa5798-16de-4e0a-9c9e-200d8099a099","Type":"ContainerDied","Data":"1cd0d2b6e933453a1c3970bc4b5f314c1fe3950154c1f8edf404a94df41d6fa6"}
Jul 23 18:48:29.898221 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.898153    2561 scope.go:117] "RemoveContainer" containerID="f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c"
Jul 23 18:48:29.907656 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.907640    2561 scope.go:117] "RemoveContainer" containerID="f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c"
Jul 23 18:48:29.907889 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:48:29.907872    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c\": container with ID starting with f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c not found: ID does not exist" containerID="f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c"
Jul 23 18:48:29.907933 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.907897    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c"} err="failed to get container status \"f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c\": rpc error: code = NotFound desc = could not find container \"f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c\": container with ID starting with f6c7889b13e44007a97256bf6dcb42b083bbd7eef10e09a77aa87b43ca3c888c not found: ID does not exist"
Jul 23 18:48:29.921569 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.921546    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"]
Jul 23 18:48:29.924726 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:29.924705    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ce7c7da-5ffbd6bdf6-7txc7"]
Jul 23 18:48:30.760616 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:30.760584    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfaa5798-16de-4e0a-9c9e-200d8099a099" path="/var/lib/kubelet/pods/cfaa5798-16de-4e0a-9c9e-200d8099a099/volumes"
Jul 23 18:48:32.086832 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:32.086788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:48:33.627186 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:33.627151    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"]
Jul 23 18:48:33.629344 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:33.629324    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4ce7c7da-5h9q7"]
Jul 23 18:48:34.761319 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:48:34.761271    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66d4a32-0457-42bd-837e-05306cd484ae" path="/var/lib/kubelet/pods/c66d4a32-0457-42bd-837e-05306cd484ae/volumes"
Jul 23 18:49:03.756929 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.756895    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"]
Jul 23 18:49:03.757523 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.757253    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66d4a32-0457-42bd-837e-05306cd484ae" containerName="revoke-keys"
Jul 23 18:49:03.757523 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.757263    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66d4a32-0457-42bd-837e-05306cd484ae" containerName="revoke-keys"
Jul 23 18:49:03.757523 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.757278    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfaa5798-16de-4e0a-9c9e-200d8099a099" containerName="maas-api"
Jul 23 18:49:03.757523 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.757313    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfaa5798-16de-4e0a-9c9e-200d8099a099" containerName="maas-api"
Jul 23 18:49:03.757523 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.757381    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66d4a32-0457-42bd-837e-05306cd484ae" containerName="revoke-keys"
Jul 23 18:49:03.757523 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.757389    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfaa5798-16de-4e0a-9c9e-200d8099a099" containerName="maas-api"
Jul 23 18:49:03.760578 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.760556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:03.763672 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.763650    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:49:03.763978 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.763959    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvms6\""
Jul 23 18:49:03.764675 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.764656    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 18:49:03.764774 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.764676    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3d6a5d58-serving-cert\""
Jul 23 18:49:03.772986 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.772967    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"]
Jul 23 18:49:03.910912 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.910868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e985b5-c293-491d-9301-758fca7703e0-openshift-service-ca\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:03.911117 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.911086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqtqk\" (UniqueName: \"kubernetes.io/projected/59e985b5-c293-491d-9301-758fca7703e0-kube-api-access-vqtqk\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:03.911302 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:03.911263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e985b5-c293-491d-9301-758fca7703e0-maas-api-tls\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:04.012218 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.012123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqtqk\" (UniqueName: \"kubernetes.io/projected/59e985b5-c293-491d-9301-758fca7703e0-kube-api-access-vqtqk\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:04.012218 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.012167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e985b5-c293-491d-9301-758fca7703e0-maas-api-tls\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:04.012218 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.012216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e985b5-c293-491d-9301-758fca7703e0-openshift-service-ca\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:04.012494 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:49:04.012379    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-serving-cert: secret "maas-api-e2e-mt-3d6a5d58-serving-cert" not found
Jul 23 18:49:04.012494 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:49:04.012453    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59e985b5-c293-491d-9301-758fca7703e0-maas-api-tls podName:59e985b5-c293-491d-9301-758fca7703e0 nodeName:}" failed. No retries permitted until 2026-07-23 18:49:04.512436679 +0000 UTC m=+1940.360127183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59e985b5-c293-491d-9301-758fca7703e0-maas-api-tls") pod "maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh" (UID: "59e985b5-c293-491d-9301-758fca7703e0") : secret "maas-api-e2e-mt-3d6a5d58-serving-cert" not found
Jul 23 18:49:04.012908 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.012889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e985b5-c293-491d-9301-758fca7703e0-openshift-service-ca\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:04.040864 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.040834    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqtqk\" (UniqueName: \"kubernetes.io/projected/59e985b5-c293-491d-9301-758fca7703e0-kube-api-access-vqtqk\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:04.516878 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.516844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e985b5-c293-491d-9301-758fca7703e0-maas-api-tls\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:04.519023 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.518998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e985b5-c293-491d-9301-758fca7703e0-maas-api-tls\") pod \"maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:04.670369 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.670332    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:04.811705 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.811678    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"]
Jul 23 18:49:04.814406 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:49:04.814375    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e985b5_c293_491d_9301_758fca7703e0.slice/crio-cfc3a20ed69aa1a70eddfb5f94f5c2b5bf7e6f4701830f434f18e8b27b79a45b WatchSource:0}: Error finding container cfc3a20ed69aa1a70eddfb5f94f5c2b5bf7e6f4701830f434f18e8b27b79a45b: Status 404 returned error can't find the container with id cfc3a20ed69aa1a70eddfb5f94f5c2b5bf7e6f4701830f434f18e8b27b79a45b
Jul 23 18:49:04.816109 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:04.816084    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:49:05.077491 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:05.077412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh" event={"ID":"59e985b5-c293-491d-9301-758fca7703e0","Type":"ContainerStarted","Data":"cfc3a20ed69aa1a70eddfb5f94f5c2b5bf7e6f4701830f434f18e8b27b79a45b"}
Jul 23 18:49:06.084579 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:06.084541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh" event={"ID":"59e985b5-c293-491d-9301-758fca7703e0","Type":"ContainerStarted","Data":"28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642"}
Jul 23 18:49:06.084971 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:06.084604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:06.103577 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:06.103530    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh" podStartSLOduration=2.68882261 podStartE2EDuration="3.103515001s" totalImagesPullingTime="414.692391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:49:06.102717752 +0000 UTC m=+1941.950408281" watchObservedRunningTime="2026-07-23 18:49:06.103515001 +0000 UTC m=+1941.951205570"
Jul 23 18:49:10.885074 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:10.885038    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:49:12.095751 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:12.095720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:49:25.187345 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:25.187308    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:49:42.079791 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:42.079712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:49:46.720414 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:46.720374    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"]
Jul 23 18:49:46.724064 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:46.724047    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"
Jul 23 18:49:46.726451 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:46.726429    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:49:46.730957 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:46.730922    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"]
Jul 23 18:49:46.884077 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:46.884042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e227f61-c49f-47e6-9a70-e8ebed92127c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv\" (UID: \"5e227f61-c49f-47e6-9a70-e8ebed92127c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"
Jul 23 18:49:46.985179 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:46.985089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e227f61-c49f-47e6-9a70-e8ebed92127c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv\" (UID: \"5e227f61-c49f-47e6-9a70-e8ebed92127c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"
Jul 23 18:49:46.985715 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:46.985694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e227f61-c49f-47e6-9a70-e8ebed92127c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv\" (UID: \"5e227f61-c49f-47e6-9a70-e8ebed92127c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"
Jul 23 18:49:47.036548 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:47.036512    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"
Jul 23 18:49:47.171831 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:47.171788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"]
Jul 23 18:49:47.175708 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:49:47.175681    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e227f61_c49f_47e6_9a70_e8ebed92127c.slice/crio-d7902a62fdea522e60e11a09fb014732716f33a4d4d9a3adbf1fcd867bef663d WatchSource:0}: Error finding container d7902a62fdea522e60e11a09fb014732716f33a4d4d9a3adbf1fcd867bef663d: Status 404 returned error can't find the container with id d7902a62fdea522e60e11a09fb014732716f33a4d4d9a3adbf1fcd867bef663d
Jul 23 18:49:47.288757 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:47.288659    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv" event={"ID":"5e227f61-c49f-47e6-9a70-e8ebed92127c","Type":"ContainerStarted","Data":"d7902a62fdea522e60e11a09fb014732716f33a4d4d9a3adbf1fcd867bef663d"}
Jul 23 18:49:48.295092 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:48.295058    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e227f61-c49f-47e6-9a70-e8ebed92127c" containerID="df1c902905fa4f1e6f8fe4a6027141a6e92528920923e1beeb592f08f6bf8a32" exitCode=0
Jul 23 18:49:48.295495 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:48.295142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv" event={"ID":"5e227f61-c49f-47e6-9a70-e8ebed92127c","Type":"ContainerDied","Data":"df1c902905fa4f1e6f8fe4a6027141a6e92528920923e1beeb592f08f6bf8a32"}
Jul 23 18:49:49.434312 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:49.434261    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"
Jul 23 18:49:49.508996 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:49.508966    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e227f61-c49f-47e6-9a70-e8ebed92127c-maas-api-service-ca\") pod \"5e227f61-c49f-47e6-9a70-e8ebed92127c\" (UID: \"5e227f61-c49f-47e6-9a70-e8ebed92127c\") "
Jul 23 18:49:49.509375 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:49.509355    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e227f61-c49f-47e6-9a70-e8ebed92127c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e227f61-c49f-47e6-9a70-e8ebed92127c" (UID: "5e227f61-c49f-47e6-9a70-e8ebed92127c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:49:49.610161 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:49.610084    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e227f61-c49f-47e6-9a70-e8ebed92127c-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:49:50.307576 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:50.307544    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"
Jul 23 18:49:50.307755 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:50.307539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv" event={"ID":"5e227f61-c49f-47e6-9a70-e8ebed92127c","Type":"ContainerDied","Data":"d7902a62fdea522e60e11a09fb014732716f33a4d4d9a3adbf1fcd867bef663d"}
Jul 23 18:49:50.307755 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:49:50.307665    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7902a62fdea522e60e11a09fb014732716f33a4d4d9a3adbf1fcd867bef663d"
Jul 23 18:50:01.276830 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:01.276793    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:50:02.314548 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.314516    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"]
Jul 23 18:50:02.314969 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.314717    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh" podUID="59e985b5-c293-491d-9301-758fca7703e0" containerName="maas-api" containerID="cri-o://28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642" gracePeriod=30
Jul 23 18:50:02.557945 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.557921    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:50:02.631874 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.631780    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqtqk\" (UniqueName: \"kubernetes.io/projected/59e985b5-c293-491d-9301-758fca7703e0-kube-api-access-vqtqk\") pod \"59e985b5-c293-491d-9301-758fca7703e0\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") "
Jul 23 18:50:02.632044 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.631871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e985b5-c293-491d-9301-758fca7703e0-maas-api-tls\") pod \"59e985b5-c293-491d-9301-758fca7703e0\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") "
Jul 23 18:50:02.632044 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.631906    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e985b5-c293-491d-9301-758fca7703e0-openshift-service-ca\") pod \"59e985b5-c293-491d-9301-758fca7703e0\" (UID: \"59e985b5-c293-491d-9301-758fca7703e0\") "
Jul 23 18:50:02.632214 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.632193    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e985b5-c293-491d-9301-758fca7703e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59e985b5-c293-491d-9301-758fca7703e0" (UID: "59e985b5-c293-491d-9301-758fca7703e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:50:02.633893 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.633865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e985b5-c293-491d-9301-758fca7703e0-kube-api-access-vqtqk" (OuterVolumeSpecName: "kube-api-access-vqtqk") pod "59e985b5-c293-491d-9301-758fca7703e0" (UID: "59e985b5-c293-491d-9301-758fca7703e0"). InnerVolumeSpecName "kube-api-access-vqtqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:50:02.633989 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.633893    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59e985b5-c293-491d-9301-758fca7703e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59e985b5-c293-491d-9301-758fca7703e0" (UID: "59e985b5-c293-491d-9301-758fca7703e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:50:02.733363 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.733332    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqtqk\" (UniqueName: \"kubernetes.io/projected/59e985b5-c293-491d-9301-758fca7703e0-kube-api-access-vqtqk\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:50:02.733363 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.733359    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59e985b5-c293-491d-9301-758fca7703e0-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:50:02.733363 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:02.733369    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59e985b5-c293-491d-9301-758fca7703e0-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:50:03.374589 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:03.374556    2561 generic.go:356] "Generic (PLEG): container finished" podID="59e985b5-c293-491d-9301-758fca7703e0" containerID="28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642" exitCode=0
Jul 23 18:50:03.375057 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:03.374597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh" event={"ID":"59e985b5-c293-491d-9301-758fca7703e0","Type":"ContainerDied","Data":"28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642"}
Jul 23 18:50:03.375057 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:03.374638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh" event={"ID":"59e985b5-c293-491d-9301-758fca7703e0","Type":"ContainerDied","Data":"cfc3a20ed69aa1a70eddfb5f94f5c2b5bf7e6f4701830f434f18e8b27b79a45b"}
Jul 23 18:50:03.375057 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:03.374661    2561 scope.go:117] "RemoveContainer" containerID="28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642"
Jul 23 18:50:03.375057 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:03.374602    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"
Jul 23 18:50:03.383570 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:03.383547    2561 scope.go:117] "RemoveContainer" containerID="28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642"
Jul 23 18:50:03.383848 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:50:03.383826    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642\": container with ID starting with 28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642 not found: ID does not exist" containerID="28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642"
Jul 23 18:50:03.383949 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:03.383856    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642"} err="failed to get container status \"28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642\": rpc error: code = NotFound desc = could not find container \"28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642\": container with ID starting with 28e601044252dc03f85261136cda282b3338663f34588ec996c73db417ddb642 not found: ID does not exist"
Jul 23 18:50:03.392276 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:03.392252    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"]
Jul 23 18:50:03.396409 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:03.396389    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d6a5d58-59df548b6-wt6kh"]
Jul 23 18:50:04.760748 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:04.760717    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59e985b5-c293-491d-9301-758fca7703e0" path="/var/lib/kubelet/pods/59e985b5-c293-491d-9301-758fca7703e0/volumes"
Jul 23 18:50:06.805118 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:06.805083    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"]
Jul 23 18:50:06.807786 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:06.807760    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d6a5d58-kddzv"]
Jul 23 18:50:08.761189 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:08.761152    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e227f61-c49f-47e6-9a70-e8ebed92127c" path="/var/lib/kubelet/pods/5e227f61-c49f-47e6-9a70-e8ebed92127c/volumes"
Jul 23 18:50:10.124259 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:10.124221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:50:44.797696 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:44.797666    2561 scope.go:117] "RemoveContainer" containerID="c30ba66678ccb2881350f5ced9e85478a6e74ae9677edb23640df5f0f81e40ed"
Jul 23 18:50:44.805680 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:50:44.805658    2561 scope.go:117] "RemoveContainer" containerID="ff1eab16b8d5033a72fea0576e9eb3eca9afe2df016b89f6b1123ad4aa1f9af7"
Jul 23 18:51:07.692776 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:51:07.692682    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:51:19.590496 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:51:19.590463    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:51:44.761160 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:51:44.761128    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:51:44.768523 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:51:44.768501    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:51:44.831372 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:51:44.831344    2561 scope.go:117] "RemoveContainer" containerID="304365a05193efd45a4063304a4a5f5626b9f7f17834addce0a65ff8fa974410"
Jul 23 18:51:44.839185 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:51:44.839163    2561 scope.go:117] "RemoveContainer" containerID="d102b9893ae8e4dfda9f23276ee58d185a35693514332993afba24a85a20a944"
Jul 23 18:52:03.640860 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.640825    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"]
Jul 23 18:52:03.641488 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.641203    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59e985b5-c293-491d-9301-758fca7703e0" containerName="maas-api"
Jul 23 18:52:03.641488 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.641213    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59e985b5-c293-491d-9301-758fca7703e0" containerName="maas-api"
Jul 23 18:52:03.641488 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.641223    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e227f61-c49f-47e6-9a70-e8ebed92127c" containerName="revoke-keys"
Jul 23 18:52:03.641488 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.641229    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e227f61-c49f-47e6-9a70-e8ebed92127c" containerName="revoke-keys"
Jul 23 18:52:03.641488 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.641297    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e227f61-c49f-47e6-9a70-e8ebed92127c" containerName="revoke-keys"
Jul 23 18:52:03.641488 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.641307    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59e985b5-c293-491d-9301-758fca7703e0" containerName="maas-api"
Jul 23 18:52:03.644351 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.644330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:03.646612 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.646590    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e16158-serving-cert\""
Jul 23 18:52:03.646734 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.646653    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:52:03.647456 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.647439    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 18:52:03.647541 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.647459    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvms6\""
Jul 23 18:52:03.656234 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.656209    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"]
Jul 23 18:52:03.707841 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.707808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006443e6-8a94-4bb2-9929-9c46fd01a95f-maas-api-tls\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:03.708032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.707949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fh45\" (UniqueName: \"kubernetes.io/projected/006443e6-8a94-4bb2-9929-9c46fd01a95f-kube-api-access-9fh45\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:03.708032 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.707982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006443e6-8a94-4bb2-9929-9c46fd01a95f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:03.809202 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.809162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fh45\" (UniqueName: \"kubernetes.io/projected/006443e6-8a94-4bb2-9929-9c46fd01a95f-kube-api-access-9fh45\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:03.809202 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.809209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006443e6-8a94-4bb2-9929-9c46fd01a95f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:03.809495 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.809253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006443e6-8a94-4bb2-9929-9c46fd01a95f-maas-api-tls\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:03.809619 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:52:03.809595    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-serving-cert: secret "maas-api-e2e-inf-a-e16158-serving-cert" not found
Jul 23 18:52:03.809692 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:52:03.809680    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/006443e6-8a94-4bb2-9929-9c46fd01a95f-maas-api-tls podName:006443e6-8a94-4bb2-9929-9c46fd01a95f nodeName:}" failed. No retries permitted until 2026-07-23 18:52:04.309656611 +0000 UTC m=+2120.157347121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/006443e6-8a94-4bb2-9929-9c46fd01a95f-maas-api-tls") pod "maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7" (UID: "006443e6-8a94-4bb2-9929-9c46fd01a95f") : secret "maas-api-e2e-inf-a-e16158-serving-cert" not found
Jul 23 18:52:03.810172 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.810152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006443e6-8a94-4bb2-9929-9c46fd01a95f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:03.819936 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:03.819904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fh45\" (UniqueName: \"kubernetes.io/projected/006443e6-8a94-4bb2-9929-9c46fd01a95f-kube-api-access-9fh45\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:04.314468 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:04.314431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006443e6-8a94-4bb2-9929-9c46fd01a95f-maas-api-tls\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:04.316846 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:04.316819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006443e6-8a94-4bb2-9929-9c46fd01a95f-maas-api-tls\") pod \"maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:04.554525 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:04.554486    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:04.694702 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:04.694674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"]
Jul 23 18:52:04.696989 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:52:04.696962    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006443e6_8a94_4bb2_9929_9c46fd01a95f.slice/crio-642724f003b0adda42efc20a7ff8f2496a6635023839f34665839cde18ad728d WatchSource:0}: Error finding container 642724f003b0adda42efc20a7ff8f2496a6635023839f34665839cde18ad728d: Status 404 returned error can't find the container with id 642724f003b0adda42efc20a7ff8f2496a6635023839f34665839cde18ad728d
Jul 23 18:52:04.985850 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:04.985815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7" event={"ID":"006443e6-8a94-4bb2-9929-9c46fd01a95f","Type":"ContainerStarted","Data":"642724f003b0adda42efc20a7ff8f2496a6635023839f34665839cde18ad728d"}
Jul 23 18:52:05.992980 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:05.992938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7" event={"ID":"006443e6-8a94-4bb2-9929-9c46fd01a95f","Type":"ContainerStarted","Data":"7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540"}
Jul 23 18:52:05.993411 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:05.993070    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:06.010821 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:06.010770    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7" podStartSLOduration=2.53615052 podStartE2EDuration="3.010756412s" totalImagesPullingTime="474.605892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:52:06.009560458 +0000 UTC m=+2121.857250998" watchObservedRunningTime="2026-07-23 18:52:06.010756412 +0000 UTC m=+2121.858446938"
Jul 23 18:52:08.986216 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:08.986184    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:52:11.887411 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:11.887370    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"]
Jul 23 18:52:11.891047 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:11.891028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:11.894443 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:11.894423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-aa7092-serving-cert\""
Jul 23 18:52:11.904787 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:11.904759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"]
Jul 23 18:52:11.983741 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:11.983710    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba72717c-dbfd-44be-abd7-bde26783ff0f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:11.983887 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:11.983758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba72717c-dbfd-44be-abd7-bde26783ff0f-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:11.983887 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:11.983843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74hd\" (UniqueName: \"kubernetes.io/projected/ba72717c-dbfd-44be-abd7-bde26783ff0f-kube-api-access-b74hd\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:12.003579 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.003548    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:52:12.084644 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.084607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba72717c-dbfd-44be-abd7-bde26783ff0f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:12.084810 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.084675    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba72717c-dbfd-44be-abd7-bde26783ff0f-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:12.084810 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:52:12.084770    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-serving-cert: secret "maas-api-e2e-inf-b-aa7092-serving-cert" not found
Jul 23 18:52:12.084935 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.084809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74hd\" (UniqueName: \"kubernetes.io/projected/ba72717c-dbfd-44be-abd7-bde26783ff0f-kube-api-access-b74hd\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:12.084935 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:52:12.084858    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba72717c-dbfd-44be-abd7-bde26783ff0f-maas-api-tls podName:ba72717c-dbfd-44be-abd7-bde26783ff0f nodeName:}" failed. No retries permitted until 2026-07-23 18:52:12.584835735 +0000 UTC m=+2128.432526254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba72717c-dbfd-44be-abd7-bde26783ff0f-maas-api-tls") pod "maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8" (UID: "ba72717c-dbfd-44be-abd7-bde26783ff0f") : secret "maas-api-e2e-inf-b-aa7092-serving-cert" not found
Jul 23 18:52:12.085274 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.085256    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba72717c-dbfd-44be-abd7-bde26783ff0f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:12.095699 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.095674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74hd\" (UniqueName: \"kubernetes.io/projected/ba72717c-dbfd-44be-abd7-bde26783ff0f-kube-api-access-b74hd\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:12.590529 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.590468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba72717c-dbfd-44be-abd7-bde26783ff0f-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:12.592946 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.592920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba72717c-dbfd-44be-abd7-bde26783ff0f-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:12.802051 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.802018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:12.939393 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:12.939295    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"]
Jul 23 18:52:12.941721 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:52:12.941698    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba72717c_dbfd_44be_abd7_bde26783ff0f.slice/crio-622fedc1d84141d41155ea6ad2740cc659319b6793218eea809ef997c56de775 WatchSource:0}: Error finding container 622fedc1d84141d41155ea6ad2740cc659319b6793218eea809ef997c56de775: Status 404 returned error can't find the container with id 622fedc1d84141d41155ea6ad2740cc659319b6793218eea809ef997c56de775
Jul 23 18:52:13.028413 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:13.028386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8" event={"ID":"ba72717c-dbfd-44be-abd7-bde26783ff0f","Type":"ContainerStarted","Data":"622fedc1d84141d41155ea6ad2740cc659319b6793218eea809ef997c56de775"}
Jul 23 18:52:14.035108 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:14.035065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8" event={"ID":"ba72717c-dbfd-44be-abd7-bde26783ff0f","Type":"ContainerStarted","Data":"22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494"}
Jul 23 18:52:14.035552 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:14.035182    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:14.052861 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:14.052818    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8" podStartSLOduration=2.714411849 podStartE2EDuration="3.05280364s" totalImagesPullingTime="338.391791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:52:14.05162675 +0000 UTC m=+2129.899317303" watchObservedRunningTime="2026-07-23 18:52:14.05280364 +0000 UTC m=+2129.900494168"
Jul 23 18:52:20.048236 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:20.048203    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:52:25.383800 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:25.383761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:52:42.092578 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:42.092491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:52:44.860816 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:44.860784    2561 scope.go:117] "RemoveContainer" containerID="afb25ea62eb10212bfc4e1d7f3ce39bf889b298ead0f0c52c5cabde6859030d9"
Jul 23 18:52:53.586486 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:52:53.586451    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:53:09.288978 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:09.288941    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:53:23.079656 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:23.079625    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:53:47.058728 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:47.058686    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"]
Jul 23 18:53:47.070051 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:47.070016    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"
Jul 23 18:53:47.071875 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:47.071845    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"]
Jul 23 18:53:47.072965 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:47.072937    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:53:47.232713 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:47.232665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b7987-ab3d-4329-b564-2c5c3057a209-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x\" (UID: \"389b7987-ab3d-4329-b564-2c5c3057a209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"
Jul 23 18:53:47.334280 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:47.334173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b7987-ab3d-4329-b564-2c5c3057a209-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x\" (UID: \"389b7987-ab3d-4329-b564-2c5c3057a209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"
Jul 23 18:53:47.334996 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:47.334972    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b7987-ab3d-4329-b564-2c5c3057a209-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x\" (UID: \"389b7987-ab3d-4329-b564-2c5c3057a209\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"
Jul 23 18:53:47.383904 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:47.383869    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"
Jul 23 18:53:47.527791 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:47.527764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"]
Jul 23 18:53:47.530014 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:53:47.529984    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389b7987_ab3d_4329_b564_2c5c3057a209.slice/crio-9d6c344b31b2ee0976fd196d3dce9906781b53486bd48bcde83fdfd44a92493a WatchSource:0}: Error finding container 9d6c344b31b2ee0976fd196d3dce9906781b53486bd48bcde83fdfd44a92493a: Status 404 returned error can't find the container with id 9d6c344b31b2ee0976fd196d3dce9906781b53486bd48bcde83fdfd44a92493a
Jul 23 18:53:48.517936 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:48.517844    2561 generic.go:356] "Generic (PLEG): container finished" podID="389b7987-ab3d-4329-b564-2c5c3057a209" containerID="bbdffae42be618d881611eeed33a3c47fb7ff72fa989e5716e23deed7f2296d4" exitCode=0
Jul 23 18:53:48.517936 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:48.517917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x" event={"ID":"389b7987-ab3d-4329-b564-2c5c3057a209","Type":"ContainerDied","Data":"bbdffae42be618d881611eeed33a3c47fb7ff72fa989e5716e23deed7f2296d4"}
Jul 23 18:53:48.518354 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:48.517957    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x" event={"ID":"389b7987-ab3d-4329-b564-2c5c3057a209","Type":"ContainerStarted","Data":"9d6c344b31b2ee0976fd196d3dce9906781b53486bd48bcde83fdfd44a92493a"}
Jul 23 18:53:49.652108 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:49.652075    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"
Jul 23 18:53:49.757885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:49.757852    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b7987-ab3d-4329-b564-2c5c3057a209-maas-api-service-ca\") pod \"389b7987-ab3d-4329-b564-2c5c3057a209\" (UID: \"389b7987-ab3d-4329-b564-2c5c3057a209\") "
Jul 23 18:53:49.758233 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:49.758209    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389b7987-ab3d-4329-b564-2c5c3057a209-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "389b7987-ab3d-4329-b564-2c5c3057a209" (UID: "389b7987-ab3d-4329-b564-2c5c3057a209"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:53:49.858777 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:49.858699    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389b7987-ab3d-4329-b564-2c5c3057a209-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:53:50.528957 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:50.528925    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"
Jul 23 18:53:50.529119 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:50.528962    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x" event={"ID":"389b7987-ab3d-4329-b564-2c5c3057a209","Type":"ContainerDied","Data":"9d6c344b31b2ee0976fd196d3dce9906781b53486bd48bcde83fdfd44a92493a"}
Jul 23 18:53:50.529119 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:50.529008    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6c344b31b2ee0976fd196d3dce9906781b53486bd48bcde83fdfd44a92493a"
Jul 23 18:53:51.080831 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:51.080801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:53:57.536509 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.536474    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"]
Jul 23 18:53:57.537026 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.536680    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7" podUID="006443e6-8a94-4bb2-9929-9c46fd01a95f" containerName="maas-api" containerID="cri-o://7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540" gracePeriod=30
Jul 23 18:53:57.786426 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.786400    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:53:57.928822 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.928743    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006443e6-8a94-4bb2-9929-9c46fd01a95f-maas-api-tls\") pod \"006443e6-8a94-4bb2-9929-9c46fd01a95f\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") "
Jul 23 18:53:57.928966 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.928839    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fh45\" (UniqueName: \"kubernetes.io/projected/006443e6-8a94-4bb2-9929-9c46fd01a95f-kube-api-access-9fh45\") pod \"006443e6-8a94-4bb2-9929-9c46fd01a95f\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") "
Jul 23 18:53:57.928966 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.928868    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006443e6-8a94-4bb2-9929-9c46fd01a95f-openshift-service-ca\") pod \"006443e6-8a94-4bb2-9929-9c46fd01a95f\" (UID: \"006443e6-8a94-4bb2-9929-9c46fd01a95f\") "
Jul 23 18:53:57.929740 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.929279    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006443e6-8a94-4bb2-9929-9c46fd01a95f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "006443e6-8a94-4bb2-9929-9c46fd01a95f" (UID: "006443e6-8a94-4bb2-9929-9c46fd01a95f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:53:57.930243 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.929970    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006443e6-8a94-4bb2-9929-9c46fd01a95f-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:53:57.931016 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.930990    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/006443e6-8a94-4bb2-9929-9c46fd01a95f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "006443e6-8a94-4bb2-9929-9c46fd01a95f" (UID: "006443e6-8a94-4bb2-9929-9c46fd01a95f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:53:57.931102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:57.931089    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006443e6-8a94-4bb2-9929-9c46fd01a95f-kube-api-access-9fh45" (OuterVolumeSpecName: "kube-api-access-9fh45") pod "006443e6-8a94-4bb2-9929-9c46fd01a95f" (UID: "006443e6-8a94-4bb2-9929-9c46fd01a95f"). InnerVolumeSpecName "kube-api-access-9fh45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:53:58.030561 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.030527    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fh45\" (UniqueName: \"kubernetes.io/projected/006443e6-8a94-4bb2-9929-9c46fd01a95f-kube-api-access-9fh45\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:53:58.030561 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.030556    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006443e6-8a94-4bb2-9929-9c46fd01a95f-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:53:58.570395 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.570358    2561 generic.go:356] "Generic (PLEG): container finished" podID="006443e6-8a94-4bb2-9929-9c46fd01a95f" containerID="7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540" exitCode=0
Jul 23 18:53:58.570849 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.570402    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"
Jul 23 18:53:58.570849 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.570430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7" event={"ID":"006443e6-8a94-4bb2-9929-9c46fd01a95f","Type":"ContainerDied","Data":"7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540"}
Jul 23 18:53:58.570849 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.570481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7" event={"ID":"006443e6-8a94-4bb2-9929-9c46fd01a95f","Type":"ContainerDied","Data":"642724f003b0adda42efc20a7ff8f2496a6635023839f34665839cde18ad728d"}
Jul 23 18:53:58.570849 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.570517    2561 scope.go:117] "RemoveContainer" containerID="7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540"
Jul 23 18:53:58.579942 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.579925    2561 scope.go:117] "RemoveContainer" containerID="7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540"
Jul 23 18:53:58.580203 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:53:58.580184    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540\": container with ID starting with 7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540 not found: ID does not exist" containerID="7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540"
Jul 23 18:53:58.580266 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.580214    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540"} err="failed to get container status \"7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540\": rpc error: code = NotFound desc = could not find container \"7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540\": container with ID starting with 7d74bd5b2b048a7ca2e9ee2ce5d89e0ffd0fd7dc9973870757c8d2808dc31540 not found: ID does not exist"
Jul 23 18:53:58.594465 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.594435    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"]
Jul 23 18:53:58.600805 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.600779    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e16158-694fbdd9f4-zljm7"]
Jul 23 18:53:58.761223 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:53:58.761182    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="006443e6-8a94-4bb2-9929-9c46fd01a95f" path="/var/lib/kubelet/pods/006443e6-8a94-4bb2-9929-9c46fd01a95f/volumes"
Jul 23 18:54:02.150925 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:02.150888    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"]
Jul 23 18:54:02.153620 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:02.153596    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e16158-nrz4x"]
Jul 23 18:54:02.761864 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:02.761831    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="389b7987-ab3d-4329-b564-2c5c3057a209" path="/var/lib/kubelet/pods/389b7987-ab3d-4329-b564-2c5c3057a209/volumes"
Jul 23 18:54:03.413829 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:03.413792    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:54:09.282203 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.282162    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"]
Jul 23 18:54:09.282706 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.282687    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="006443e6-8a94-4bb2-9929-9c46fd01a95f" containerName="maas-api"
Jul 23 18:54:09.282762 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.282711    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="006443e6-8a94-4bb2-9929-9c46fd01a95f" containerName="maas-api"
Jul 23 18:54:09.282762 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.282724    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="389b7987-ab3d-4329-b564-2c5c3057a209" containerName="revoke-keys"
Jul 23 18:54:09.282762 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.282733    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="389b7987-ab3d-4329-b564-2c5c3057a209" containerName="revoke-keys"
Jul 23 18:54:09.282877 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.282864    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="389b7987-ab3d-4329-b564-2c5c3057a209" containerName="revoke-keys"
Jul 23 18:54:09.282913 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.282882    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="006443e6-8a94-4bb2-9929-9c46fd01a95f" containerName="maas-api"
Jul 23 18:54:09.287560 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.287540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"
Jul 23 18:54:09.292247 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.292222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:54:09.294329 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.294307    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"]
Jul 23 18:54:09.324767 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.324740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0641f68-9dd5-43a8-aa30-8fdbf195f020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-aa7092-br46f\" (UID: \"d0641f68-9dd5-43a8-aa30-8fdbf195f020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"
Jul 23 18:54:09.426049 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.426018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0641f68-9dd5-43a8-aa30-8fdbf195f020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-aa7092-br46f\" (UID: \"d0641f68-9dd5-43a8-aa30-8fdbf195f020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"
Jul 23 18:54:09.426665 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.426647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0641f68-9dd5-43a8-aa30-8fdbf195f020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-aa7092-br46f\" (UID: \"d0641f68-9dd5-43a8-aa30-8fdbf195f020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"
Jul 23 18:54:09.600899 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.600814    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"
Jul 23 18:54:09.735409 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:09.735374    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"]
Jul 23 18:54:09.738367 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:54:09.738340    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0641f68_9dd5_43a8_aa30_8fdbf195f020.slice/crio-9ec0135f488562c81ecc4922f90a813c73d2055f6d81a43306ed0a05357f0e99 WatchSource:0}: Error finding container 9ec0135f488562c81ecc4922f90a813c73d2055f6d81a43306ed0a05357f0e99: Status 404 returned error can't find the container with id 9ec0135f488562c81ecc4922f90a813c73d2055f6d81a43306ed0a05357f0e99
Jul 23 18:54:10.631314 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:10.631264    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0641f68-9dd5-43a8-aa30-8fdbf195f020" containerID="a0e647e9452efcd2c19f553b6f7996316e4c33fb73109a8eac3c1ead1790a2b4" exitCode=0
Jul 23 18:54:10.631686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:10.631346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f" event={"ID":"d0641f68-9dd5-43a8-aa30-8fdbf195f020","Type":"ContainerDied","Data":"a0e647e9452efcd2c19f553b6f7996316e4c33fb73109a8eac3c1ead1790a2b4"}
Jul 23 18:54:10.631686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:10.631384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f" event={"ID":"d0641f68-9dd5-43a8-aa30-8fdbf195f020","Type":"ContainerStarted","Data":"9ec0135f488562c81ecc4922f90a813c73d2055f6d81a43306ed0a05357f0e99"}
Jul 23 18:54:11.762174 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:11.762151    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"
Jul 23 18:54:11.845424 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:11.845388    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0641f68-9dd5-43a8-aa30-8fdbf195f020-maas-api-service-ca\") pod \"d0641f68-9dd5-43a8-aa30-8fdbf195f020\" (UID: \"d0641f68-9dd5-43a8-aa30-8fdbf195f020\") "
Jul 23 18:54:11.845784 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:11.845760    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0641f68-9dd5-43a8-aa30-8fdbf195f020-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0641f68-9dd5-43a8-aa30-8fdbf195f020" (UID: "d0641f68-9dd5-43a8-aa30-8fdbf195f020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:54:11.946619 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:11.946577    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0641f68-9dd5-43a8-aa30-8fdbf195f020-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:54:12.642301 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:12.642259    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"
Jul 23 18:54:12.642442 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:12.642255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f" event={"ID":"d0641f68-9dd5-43a8-aa30-8fdbf195f020","Type":"ContainerDied","Data":"9ec0135f488562c81ecc4922f90a813c73d2055f6d81a43306ed0a05357f0e99"}
Jul 23 18:54:12.642442 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:12.642416    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ec0135f488562c81ecc4922f90a813c73d2055f6d81a43306ed0a05357f0e99"
Jul 23 18:54:19.709808 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:19.709768    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"]
Jul 23 18:54:19.710310 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:19.710090    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8" podUID="ba72717c-dbfd-44be-abd7-bde26783ff0f" containerName="maas-api" containerID="cri-o://22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494" gracePeriod=30
Jul 23 18:54:19.962139 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:19.962084    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:54:20.020682 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.020650    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b74hd\" (UniqueName: \"kubernetes.io/projected/ba72717c-dbfd-44be-abd7-bde26783ff0f-kube-api-access-b74hd\") pod \"ba72717c-dbfd-44be-abd7-bde26783ff0f\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") "
Jul 23 18:54:20.020830 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.020730    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba72717c-dbfd-44be-abd7-bde26783ff0f-maas-api-tls\") pod \"ba72717c-dbfd-44be-abd7-bde26783ff0f\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") "
Jul 23 18:54:20.020830 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.020776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba72717c-dbfd-44be-abd7-bde26783ff0f-openshift-service-ca\") pod \"ba72717c-dbfd-44be-abd7-bde26783ff0f\" (UID: \"ba72717c-dbfd-44be-abd7-bde26783ff0f\") "
Jul 23 18:54:20.021172 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.021150    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba72717c-dbfd-44be-abd7-bde26783ff0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba72717c-dbfd-44be-abd7-bde26783ff0f" (UID: "ba72717c-dbfd-44be-abd7-bde26783ff0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:54:20.022681 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.022658    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba72717c-dbfd-44be-abd7-bde26783ff0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba72717c-dbfd-44be-abd7-bde26783ff0f" (UID: "ba72717c-dbfd-44be-abd7-bde26783ff0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:54:20.022772 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.022658    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba72717c-dbfd-44be-abd7-bde26783ff0f-kube-api-access-b74hd" (OuterVolumeSpecName: "kube-api-access-b74hd") pod "ba72717c-dbfd-44be-abd7-bde26783ff0f" (UID: "ba72717c-dbfd-44be-abd7-bde26783ff0f"). InnerVolumeSpecName "kube-api-access-b74hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:54:20.121674 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.121637    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74hd\" (UniqueName: \"kubernetes.io/projected/ba72717c-dbfd-44be-abd7-bde26783ff0f-kube-api-access-b74hd\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:54:20.121674 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.121668    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba72717c-dbfd-44be-abd7-bde26783ff0f-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:54:20.121674 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.121681    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba72717c-dbfd-44be-abd7-bde26783ff0f-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:54:20.682107 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.682071    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba72717c-dbfd-44be-abd7-bde26783ff0f" containerID="22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494" exitCode=0
Jul 23 18:54:20.682267 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.682114    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"
Jul 23 18:54:20.682267 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.682153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8" event={"ID":"ba72717c-dbfd-44be-abd7-bde26783ff0f","Type":"ContainerDied","Data":"22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494"}
Jul 23 18:54:20.682267 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.682204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8" event={"ID":"ba72717c-dbfd-44be-abd7-bde26783ff0f","Type":"ContainerDied","Data":"622fedc1d84141d41155ea6ad2740cc659319b6793218eea809ef997c56de775"}
Jul 23 18:54:20.682267 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.682225    2561 scope.go:117] "RemoveContainer" containerID="22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494"
Jul 23 18:54:20.691594 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.691577    2561 scope.go:117] "RemoveContainer" containerID="22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494"
Jul 23 18:54:20.691869 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:54:20.691846    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494\": container with ID starting with 22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494 not found: ID does not exist" containerID="22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494"
Jul 23 18:54:20.691955 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.691875    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494"} err="failed to get container status \"22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494\": rpc error: code = NotFound desc = could not find container \"22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494\": container with ID starting with 22fa1329f5a3588b64bfdd932a45eec95d360ed44479c345e8c60e673d32d494 not found: ID does not exist"
Jul 23 18:54:20.703439 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.703416    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"]
Jul 23 18:54:20.707195 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.707172    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa7092-6d5669bdd5-zxpz8"]
Jul 23 18:54:20.760548 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:20.760511    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba72717c-dbfd-44be-abd7-bde26783ff0f" path="/var/lib/kubelet/pods/ba72717c-dbfd-44be-abd7-bde26783ff0f/volumes"
Jul 23 18:54:23.083677 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:23.083648    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:54:24.364271 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:24.364235    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"]
Jul 23 18:54:24.366930 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:24.366908    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aa7092-br46f"]
Jul 23 18:54:24.763195 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:24.763151    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0641f68-9dd5-43a8-aa30-8fdbf195f020" path="/var/lib/kubelet/pods/d0641f68-9dd5-43a8-aa30-8fdbf195f020/volumes"
Jul 23 18:54:35.458939 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.458904    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"]
Jul 23 18:54:35.459515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.459345    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba72717c-dbfd-44be-abd7-bde26783ff0f" containerName="maas-api"
Jul 23 18:54:35.459515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.459357    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba72717c-dbfd-44be-abd7-bde26783ff0f" containerName="maas-api"
Jul 23 18:54:35.459515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.459380    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0641f68-9dd5-43a8-aa30-8fdbf195f020" containerName="revoke-keys"
Jul 23 18:54:35.459515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.459389    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0641f68-9dd5-43a8-aa30-8fdbf195f020" containerName="revoke-keys"
Jul 23 18:54:35.459515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.459445    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba72717c-dbfd-44be-abd7-bde26783ff0f" containerName="maas-api"
Jul 23 18:54:35.459515 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.459452    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0641f68-9dd5-43a8-aa30-8fdbf195f020" containerName="revoke-keys"
Jul 23 18:54:35.463699 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.463679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:35.467517 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.467492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-04f83f-serving-cert\""
Jul 23 18:54:35.467658 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.467503    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 18:54:35.467658 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.467504    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:54:35.467871 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.467855    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvms6\""
Jul 23 18:54:35.475519 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.475493    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"]
Jul 23 18:54:35.563239 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.563205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410869b3-0d2a-44c4-b8a0-3981438b1810-maas-api-tls\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:35.563239 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.563237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410869b3-0d2a-44c4-b8a0-3981438b1810-openshift-service-ca\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:35.563511 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.563373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbv5\" (UniqueName: \"kubernetes.io/projected/410869b3-0d2a-44c4-b8a0-3981438b1810-kube-api-access-qkbv5\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:35.664618 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.664585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbv5\" (UniqueName: \"kubernetes.io/projected/410869b3-0d2a-44c4-b8a0-3981438b1810-kube-api-access-qkbv5\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:35.664807 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.664657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410869b3-0d2a-44c4-b8a0-3981438b1810-maas-api-tls\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:35.664807 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.664683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410869b3-0d2a-44c4-b8a0-3981438b1810-openshift-service-ca\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:35.664807 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:54:35.664787    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-serving-cert: secret "maas-api-e2e-api-a-04f83f-serving-cert" not found
Jul 23 18:54:35.664979 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:54:35.664845    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410869b3-0d2a-44c4-b8a0-3981438b1810-maas-api-tls podName:410869b3-0d2a-44c4-b8a0-3981438b1810 nodeName:}" failed. No retries permitted until 2026-07-23 18:54:36.164828299 +0000 UTC m=+2272.012518805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/410869b3-0d2a-44c4-b8a0-3981438b1810-maas-api-tls") pod "maas-api-e2e-api-a-04f83f-7699b79546-j6pzn" (UID: "410869b3-0d2a-44c4-b8a0-3981438b1810") : secret "maas-api-e2e-api-a-04f83f-serving-cert" not found
Jul 23 18:54:35.665374 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.665353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410869b3-0d2a-44c4-b8a0-3981438b1810-openshift-service-ca\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:35.674422 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:35.674390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbv5\" (UniqueName: \"kubernetes.io/projected/410869b3-0d2a-44c4-b8a0-3981438b1810-kube-api-access-qkbv5\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:36.169861 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:36.169825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410869b3-0d2a-44c4-b8a0-3981438b1810-maas-api-tls\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:36.172275 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:36.172248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410869b3-0d2a-44c4-b8a0-3981438b1810-maas-api-tls\") pod \"maas-api-e2e-api-a-04f83f-7699b79546-j6pzn\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:36.376928 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:36.376888    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:36.509663 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:36.509634    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"]
Jul 23 18:54:36.512504 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:54:36.512472    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410869b3_0d2a_44c4_b8a0_3981438b1810.slice/crio-dea0b8bfbd4eeb71014d8bc142e55f25515646cc9ea120e65bfaf560aa3a7146 WatchSource:0}: Error finding container dea0b8bfbd4eeb71014d8bc142e55f25515646cc9ea120e65bfaf560aa3a7146: Status 404 returned error can't find the container with id dea0b8bfbd4eeb71014d8bc142e55f25515646cc9ea120e65bfaf560aa3a7146
Jul 23 18:54:36.513795 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:36.513775    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 18:54:36.762121 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:36.762089    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn" event={"ID":"410869b3-0d2a-44c4-b8a0-3981438b1810","Type":"ContainerStarted","Data":"dea0b8bfbd4eeb71014d8bc142e55f25515646cc9ea120e65bfaf560aa3a7146"}
Jul 23 18:54:37.768831 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:37.768790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn" event={"ID":"410869b3-0d2a-44c4-b8a0-3981438b1810","Type":"ContainerStarted","Data":"80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7"}
Jul 23 18:54:37.769273 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:37.768943    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:37.787941 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:37.787897    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn" podStartSLOduration=2.32945043 podStartE2EDuration="2.787883376s" totalImagesPullingTime="458.432946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:54:37.786007184 +0000 UTC m=+2273.633697713" watchObservedRunningTime="2026-07-23 18:54:37.787883376 +0000 UTC m=+2273.635573903"
Jul 23 18:54:42.482953 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:42.482921    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:54:43.393068 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.393036    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"]
Jul 23 18:54:43.396813 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.396783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:43.400739 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.400715    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-65b807-serving-cert\""
Jul 23 18:54:43.414265 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.414236    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"]
Jul 23 18:54:43.541463 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.541423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8100815e-80a2-42ab-a0f8-da8c0eb77af5-maas-api-tls\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:43.541855 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.541489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8100815e-80a2-42ab-a0f8-da8c0eb77af5-openshift-service-ca\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:43.541855 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.541509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rxs\" (UniqueName: \"kubernetes.io/projected/8100815e-80a2-42ab-a0f8-da8c0eb77af5-kube-api-access-75rxs\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:43.642709 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.642678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8100815e-80a2-42ab-a0f8-da8c0eb77af5-maas-api-tls\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:43.642873 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.642737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8100815e-80a2-42ab-a0f8-da8c0eb77af5-openshift-service-ca\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:43.642873 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:54:43.642825    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-serving-cert: secret "maas-api-e2e-api-b-65b807-serving-cert" not found
Jul 23 18:54:43.642873 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.642842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rxs\" (UniqueName: \"kubernetes.io/projected/8100815e-80a2-42ab-a0f8-da8c0eb77af5-kube-api-access-75rxs\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:43.643115 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:54:43.642894    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8100815e-80a2-42ab-a0f8-da8c0eb77af5-maas-api-tls podName:8100815e-80a2-42ab-a0f8-da8c0eb77af5 nodeName:}" failed. No retries permitted until 2026-07-23 18:54:44.142874539 +0000 UTC m=+2279.990565060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8100815e-80a2-42ab-a0f8-da8c0eb77af5-maas-api-tls") pod "maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s" (UID: "8100815e-80a2-42ab-a0f8-da8c0eb77af5") : secret "maas-api-e2e-api-b-65b807-serving-cert" not found
Jul 23 18:54:43.643359 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.643341    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8100815e-80a2-42ab-a0f8-da8c0eb77af5-openshift-service-ca\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:43.652725 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.652701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rxs\" (UniqueName: \"kubernetes.io/projected/8100815e-80a2-42ab-a0f8-da8c0eb77af5-kube-api-access-75rxs\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:43.778432 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:43.778403    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:54:44.147145 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:44.147104    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8100815e-80a2-42ab-a0f8-da8c0eb77af5-maas-api-tls\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:44.149488 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:44.149460    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8100815e-80a2-42ab-a0f8-da8c0eb77af5-maas-api-tls\") pod \"maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:44.307630 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:44.307593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:44.444670 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:44.444646    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"]
Jul 23 18:54:44.447640 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:54:44.447605    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8100815e_80a2_42ab_a0f8_da8c0eb77af5.slice/crio-d444b8a59cf9112e1ca6eab8d27d4bd74899531e78ed74fa849c8e201cfc3cbb WatchSource:0}: Error finding container d444b8a59cf9112e1ca6eab8d27d4bd74899531e78ed74fa849c8e201cfc3cbb: Status 404 returned error can't find the container with id d444b8a59cf9112e1ca6eab8d27d4bd74899531e78ed74fa849c8e201cfc3cbb
Jul 23 18:54:44.804101 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:44.804063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s" event={"ID":"8100815e-80a2-42ab-a0f8-da8c0eb77af5","Type":"ContainerStarted","Data":"d444b8a59cf9112e1ca6eab8d27d4bd74899531e78ed74fa849c8e201cfc3cbb"}
Jul 23 18:54:44.891575 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:44.888531    2561 scope.go:117] "RemoveContainer" containerID="e44a570809b2e0b378473043e1b0efa45fefcf57acae9ee6284e116bbc4fcf23"
Jul 23 18:54:45.810317 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:45.810264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s" event={"ID":"8100815e-80a2-42ab-a0f8-da8c0eb77af5","Type":"ContainerStarted","Data":"de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039"}
Jul 23 18:54:45.810724 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:45.810345    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:45.828304 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:45.828233    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s" podStartSLOduration=2.412906859 podStartE2EDuration="2.828205212s" totalImagesPullingTime="415.298353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:54:45.826865967 +0000 UTC m=+2281.674556505" watchObservedRunningTime="2026-07-23 18:54:45.828205212 +0000 UTC m=+2281.675895739"
Jul 23 18:54:51.820100 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:51.820052    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:54:55.784649 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:55.784611    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"]
Jul 23 18:54:55.788029 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:55.788013    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"
Jul 23 18:54:55.790548 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:55.790526    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:54:55.795710 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:55.795246    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"]
Jul 23 18:54:55.840551 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:55.840524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fff6708-718a-4489-8425-3112cbee7a49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs\" (UID: \"8fff6708-718a-4489-8425-3112cbee7a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"
Jul 23 18:54:55.941587 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:55.941557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fff6708-718a-4489-8425-3112cbee7a49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs\" (UID: \"8fff6708-718a-4489-8425-3112cbee7a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"
Jul 23 18:54:55.942206 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:55.942181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fff6708-718a-4489-8425-3112cbee7a49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs\" (UID: \"8fff6708-718a-4489-8425-3112cbee7a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"
Jul 23 18:54:56.099305 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:56.099203    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"
Jul 23 18:54:56.247753 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:56.247734    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"]
Jul 23 18:54:56.250024 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:54:56.249994    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fff6708_718a_4489_8425_3112cbee7a49.slice/crio-68f6c915aeee50079b9eebbfa3de86650d8e74b973325262046b9eab2eb92712 WatchSource:0}: Error finding container 68f6c915aeee50079b9eebbfa3de86650d8e74b973325262046b9eab2eb92712: Status 404 returned error can't find the container with id 68f6c915aeee50079b9eebbfa3de86650d8e74b973325262046b9eab2eb92712
Jul 23 18:54:56.867422 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:56.867391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs" event={"ID":"8fff6708-718a-4489-8425-3112cbee7a49","Type":"ContainerStarted","Data":"8757b879b89bbb8c5f69e3a393dca6004be35dff0dbe95554d9499b24309fa49"}
Jul 23 18:54:56.867809 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:56.867444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs" event={"ID":"8fff6708-718a-4489-8425-3112cbee7a49","Type":"ContainerStarted","Data":"68f6c915aeee50079b9eebbfa3de86650d8e74b973325262046b9eab2eb92712"}
Jul 23 18:54:56.885482 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:56.885425    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs" podStartSLOduration=1.8854112 podStartE2EDuration="1.8854112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:54:56.884007012 +0000 UTC m=+2292.731697539" watchObservedRunningTime="2026-07-23 18:54:56.8854112 +0000 UTC m=+2292.733101726"
Jul 23 18:54:57.873663 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:57.873630    2561 generic.go:356] "Generic (PLEG): container finished" podID="8fff6708-718a-4489-8425-3112cbee7a49" containerID="8757b879b89bbb8c5f69e3a393dca6004be35dff0dbe95554d9499b24309fa49" exitCode=0
Jul 23 18:54:57.874042 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:57.873719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs" event={"ID":"8fff6708-718a-4489-8425-3112cbee7a49","Type":"ContainerDied","Data":"8757b879b89bbb8c5f69e3a393dca6004be35dff0dbe95554d9499b24309fa49"}
Jul 23 18:54:57.880846 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:57.880824    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:54:59.010457 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:59.010432    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"
Jul 23 18:54:59.067885 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:59.067857    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fff6708-718a-4489-8425-3112cbee7a49-maas-api-service-ca\") pod \"8fff6708-718a-4489-8425-3112cbee7a49\" (UID: \"8fff6708-718a-4489-8425-3112cbee7a49\") "
Jul 23 18:54:59.068211 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:59.068192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fff6708-718a-4489-8425-3112cbee7a49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8fff6708-718a-4489-8425-3112cbee7a49" (UID: "8fff6708-718a-4489-8425-3112cbee7a49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:54:59.169392 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:59.169314    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fff6708-718a-4489-8425-3112cbee7a49-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:54:59.884688 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:59.884657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs" event={"ID":"8fff6708-718a-4489-8425-3112cbee7a49","Type":"ContainerDied","Data":"68f6c915aeee50079b9eebbfa3de86650d8e74b973325262046b9eab2eb92712"}
Jul 23 18:54:59.884865 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:59.884705    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68f6c915aeee50079b9eebbfa3de86650d8e74b973325262046b9eab2eb92712"
Jul 23 18:54:59.884865 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:54:59.884669    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"
Jul 23 18:55:11.572239 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.572207    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"]
Jul 23 18:55:11.572732 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.572437    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn" podUID="410869b3-0d2a-44c4-b8a0-3981438b1810" containerName="maas-api" containerID="cri-o://80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7" gracePeriod=30
Jul 23 18:55:11.817097 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.817069    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:55:11.889113 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.889017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410869b3-0d2a-44c4-b8a0-3981438b1810-maas-api-tls\") pod \"410869b3-0d2a-44c4-b8a0-3981438b1810\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") "
Jul 23 18:55:11.889314 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.889194    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkbv5\" (UniqueName: \"kubernetes.io/projected/410869b3-0d2a-44c4-b8a0-3981438b1810-kube-api-access-qkbv5\") pod \"410869b3-0d2a-44c4-b8a0-3981438b1810\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") "
Jul 23 18:55:11.889314 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.889235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410869b3-0d2a-44c4-b8a0-3981438b1810-openshift-service-ca\") pod \"410869b3-0d2a-44c4-b8a0-3981438b1810\" (UID: \"410869b3-0d2a-44c4-b8a0-3981438b1810\") "
Jul 23 18:55:11.889691 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.889662    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410869b3-0d2a-44c4-b8a0-3981438b1810-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "410869b3-0d2a-44c4-b8a0-3981438b1810" (UID: "410869b3-0d2a-44c4-b8a0-3981438b1810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:55:11.891138 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.891106    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/410869b3-0d2a-44c4-b8a0-3981438b1810-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "410869b3-0d2a-44c4-b8a0-3981438b1810" (UID: "410869b3-0d2a-44c4-b8a0-3981438b1810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:55:11.891232 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.891169    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410869b3-0d2a-44c4-b8a0-3981438b1810-kube-api-access-qkbv5" (OuterVolumeSpecName: "kube-api-access-qkbv5") pod "410869b3-0d2a-44c4-b8a0-3981438b1810" (UID: "410869b3-0d2a-44c4-b8a0-3981438b1810"). InnerVolumeSpecName "kube-api-access-qkbv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:55:11.948789 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.948756    2561 generic.go:356] "Generic (PLEG): container finished" podID="410869b3-0d2a-44c4-b8a0-3981438b1810" containerID="80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7" exitCode=0
Jul 23 18:55:11.948789 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.948791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn" event={"ID":"410869b3-0d2a-44c4-b8a0-3981438b1810","Type":"ContainerDied","Data":"80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7"}
Jul 23 18:55:11.949002 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.948798    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"
Jul 23 18:55:11.949002 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.948827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn" event={"ID":"410869b3-0d2a-44c4-b8a0-3981438b1810","Type":"ContainerDied","Data":"dea0b8bfbd4eeb71014d8bc142e55f25515646cc9ea120e65bfaf560aa3a7146"}
Jul 23 18:55:11.949002 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.948849    2561 scope.go:117] "RemoveContainer" containerID="80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7"
Jul 23 18:55:11.957941 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.957924    2561 scope.go:117] "RemoveContainer" containerID="80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7"
Jul 23 18:55:11.958195 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:55:11.958178    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7\": container with ID starting with 80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7 not found: ID does not exist" containerID="80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7"
Jul 23 18:55:11.958264 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.958206    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7"} err="failed to get container status \"80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7\": rpc error: code = NotFound desc = could not find container \"80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7\": container with ID starting with 80925fdd04c54d185ace7c63ea51b1d5ef66e714d02310aaf8ef433c3c8a12f7 not found: ID does not exist"
Jul 23 18:55:11.969102 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.969081    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"]
Jul 23 18:55:11.973058 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.973037    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-04f83f-7699b79546-j6pzn"]
Jul 23 18:55:11.990013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.989988    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkbv5\" (UniqueName: \"kubernetes.io/projected/410869b3-0d2a-44c4-b8a0-3981438b1810-kube-api-access-qkbv5\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:55:11.990013 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.990010    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/410869b3-0d2a-44c4-b8a0-3981438b1810-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:55:11.990136 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:11.990021    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/410869b3-0d2a-44c4-b8a0-3981438b1810-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:55:12.762021 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:12.761987    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="410869b3-0d2a-44c4-b8a0-3981438b1810" path="/var/lib/kubelet/pods/410869b3-0d2a-44c4-b8a0-3981438b1810/volumes"
Jul 23 18:55:15.385943 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:15.385906    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:55:15.872015 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:15.871978    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"]
Jul 23 18:55:15.876031 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:15.876002    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f83f-dxkhs"]
Jul 23 18:55:16.760853 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:16.760806    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fff6708-718a-4489-8425-3112cbee7a49" path="/var/lib/kubelet/pods/8fff6708-718a-4489-8425-3112cbee7a49/volumes"
Jul 23 18:55:22.722753 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.722718    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"]
Jul 23 18:55:22.723350 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.723331    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410869b3-0d2a-44c4-b8a0-3981438b1810" containerName="maas-api"
Jul 23 18:55:22.723350 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.723352    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="410869b3-0d2a-44c4-b8a0-3981438b1810" containerName="maas-api"
Jul 23 18:55:22.723510 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.723379    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fff6708-718a-4489-8425-3112cbee7a49" containerName="revoke-keys"
Jul 23 18:55:22.723510 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.723388    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fff6708-718a-4489-8425-3112cbee7a49" containerName="revoke-keys"
Jul 23 18:55:22.723510 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.723494    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fff6708-718a-4489-8425-3112cbee7a49" containerName="revoke-keys"
Jul 23 18:55:22.723668 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.723512    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="410869b3-0d2a-44c4-b8a0-3981438b1810" containerName="maas-api"
Jul 23 18:55:22.726611 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.726590    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"
Jul 23 18:55:22.729086 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.729065    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 18:55:22.735103 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.735078    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"]
Jul 23 18:55:22.887554 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.887506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a1aef0-b76b-47f4-8425-67443d3098ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65b807-6mnk2\" (UID: \"90a1aef0-b76b-47f4-8425-67443d3098ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"
Jul 23 18:55:22.988399 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.988305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a1aef0-b76b-47f4-8425-67443d3098ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65b807-6mnk2\" (UID: \"90a1aef0-b76b-47f4-8425-67443d3098ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"
Jul 23 18:55:22.988944 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:22.988923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a1aef0-b76b-47f4-8425-67443d3098ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65b807-6mnk2\" (UID: \"90a1aef0-b76b-47f4-8425-67443d3098ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"
Jul 23 18:55:23.038602 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:23.038573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"
Jul 23 18:55:23.171335 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:23.171277    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"]
Jul 23 18:55:23.174375 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:55:23.174333    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a1aef0_b76b_47f4_8425_67443d3098ac.slice/crio-e953c82f4de65b209d7983b89266370a30ffd8feb5f7834a0d2d02937af1c445 WatchSource:0}: Error finding container e953c82f4de65b209d7983b89266370a30ffd8feb5f7834a0d2d02937af1c445: Status 404 returned error can't find the container with id e953c82f4de65b209d7983b89266370a30ffd8feb5f7834a0d2d02937af1c445
Jul 23 18:55:24.011848 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:24.011811    2561 generic.go:356] "Generic (PLEG): container finished" podID="90a1aef0-b76b-47f4-8425-67443d3098ac" containerID="c4a7caf8be2f358cf0471f13180ca02d9e6c6931e604128b4a33f8b03e34c190" exitCode=0
Jul 23 18:55:24.012310 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:24.011847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2" event={"ID":"90a1aef0-b76b-47f4-8425-67443d3098ac","Type":"ContainerDied","Data":"c4a7caf8be2f358cf0471f13180ca02d9e6c6931e604128b4a33f8b03e34c190"}
Jul 23 18:55:24.012310 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:24.011905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2" event={"ID":"90a1aef0-b76b-47f4-8425-67443d3098ac","Type":"ContainerStarted","Data":"e953c82f4de65b209d7983b89266370a30ffd8feb5f7834a0d2d02937af1c445"}
Jul 23 18:55:25.145029 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:25.145003    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"
Jul 23 18:55:25.309797 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:25.309708    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a1aef0-b76b-47f4-8425-67443d3098ac-maas-api-service-ca\") pod \"90a1aef0-b76b-47f4-8425-67443d3098ac\" (UID: \"90a1aef0-b76b-47f4-8425-67443d3098ac\") "
Jul 23 18:55:25.310109 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:25.310089    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a1aef0-b76b-47f4-8425-67443d3098ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90a1aef0-b76b-47f4-8425-67443d3098ac" (UID: "90a1aef0-b76b-47f4-8425-67443d3098ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:55:25.410981 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:25.410945    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90a1aef0-b76b-47f4-8425-67443d3098ac-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:55:26.022115 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:26.022085    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"
Jul 23 18:55:26.022115 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:26.022093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2" event={"ID":"90a1aef0-b76b-47f4-8425-67443d3098ac","Type":"ContainerDied","Data":"e953c82f4de65b209d7983b89266370a30ffd8feb5f7834a0d2d02937af1c445"}
Jul 23 18:55:26.022409 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:26.022136    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e953c82f4de65b209d7983b89266370a30ffd8feb5f7834a0d2d02937af1c445"
Jul 23 18:55:38.334915 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.334831    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"]
Jul 23 18:55:38.335330 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.335086    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s" podUID="8100815e-80a2-42ab-a0f8-da8c0eb77af5" containerName="maas-api" containerID="cri-o://de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039" gracePeriod=30
Jul 23 18:55:38.575974 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.575947    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:55:38.628349 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.628261    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8100815e-80a2-42ab-a0f8-da8c0eb77af5-maas-api-tls\") pod \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") "
Jul 23 18:55:38.628349 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.628337    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8100815e-80a2-42ab-a0f8-da8c0eb77af5-openshift-service-ca\") pod \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") "
Jul 23 18:55:38.628518 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.628417    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75rxs\" (UniqueName: \"kubernetes.io/projected/8100815e-80a2-42ab-a0f8-da8c0eb77af5-kube-api-access-75rxs\") pod \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\" (UID: \"8100815e-80a2-42ab-a0f8-da8c0eb77af5\") "
Jul 23 18:55:38.628731 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.628698    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8100815e-80a2-42ab-a0f8-da8c0eb77af5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8100815e-80a2-42ab-a0f8-da8c0eb77af5" (UID: "8100815e-80a2-42ab-a0f8-da8c0eb77af5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 18:55:38.630360 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.630330    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8100815e-80a2-42ab-a0f8-da8c0eb77af5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8100815e-80a2-42ab-a0f8-da8c0eb77af5" (UID: "8100815e-80a2-42ab-a0f8-da8c0eb77af5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 18:55:38.630360 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.630344    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8100815e-80a2-42ab-a0f8-da8c0eb77af5-kube-api-access-75rxs" (OuterVolumeSpecName: "kube-api-access-75rxs") pod "8100815e-80a2-42ab-a0f8-da8c0eb77af5" (UID: "8100815e-80a2-42ab-a0f8-da8c0eb77af5"). InnerVolumeSpecName "kube-api-access-75rxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 18:55:38.729582 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.729549    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rxs\" (UniqueName: \"kubernetes.io/projected/8100815e-80a2-42ab-a0f8-da8c0eb77af5-kube-api-access-75rxs\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:55:38.729582 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.729578    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8100815e-80a2-42ab-a0f8-da8c0eb77af5-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:55:38.729582 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:38.729588    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8100815e-80a2-42ab-a0f8-da8c0eb77af5-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 18:55:39.086986 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:39.086950    2561 generic.go:356] "Generic (PLEG): container finished" podID="8100815e-80a2-42ab-a0f8-da8c0eb77af5" containerID="de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039" exitCode=0
Jul 23 18:55:39.087163 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:39.086994    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"
Jul 23 18:55:39.087163 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:39.087030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s" event={"ID":"8100815e-80a2-42ab-a0f8-da8c0eb77af5","Type":"ContainerDied","Data":"de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039"}
Jul 23 18:55:39.087163 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:39.087078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s" event={"ID":"8100815e-80a2-42ab-a0f8-da8c0eb77af5","Type":"ContainerDied","Data":"d444b8a59cf9112e1ca6eab8d27d4bd74899531e78ed74fa849c8e201cfc3cbb"}
Jul 23 18:55:39.087163 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:39.087101    2561 scope.go:117] "RemoveContainer" containerID="de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039"
Jul 23 18:55:39.095552 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:39.095529    2561 scope.go:117] "RemoveContainer" containerID="de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039"
Jul 23 18:55:39.095811 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:55:39.095793    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039\": container with ID starting with de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039 not found: ID does not exist" containerID="de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039"
Jul 23 18:55:39.095874 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:39.095824    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039"} err="failed to get container status \"de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039\": rpc error: code = NotFound desc = could not find container \"de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039\": container with ID starting with de9ae71ee86d865ffa0f3a2911998550c6b291e79bda8a4f0dbd0fb106a93039 not found: ID does not exist"
Jul 23 18:55:39.105177 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:39.105155    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"]
Jul 23 18:55:39.109362 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:39.109342    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65b807-9d6b6fcc6-7z97s"]
Jul 23 18:55:40.762127 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:40.762095    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8100815e-80a2-42ab-a0f8-da8c0eb77af5" path="/var/lib/kubelet/pods/8100815e-80a2-42ab-a0f8-da8c0eb77af5/volumes"
Jul 23 18:55:42.781686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:42.781650    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:55:42.800339 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:42.800308    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"]
Jul 23 18:55:42.802806 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:42.802784    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65b807-6mnk2"]
Jul 23 18:55:44.761158 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:44.761125    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a1aef0-b76b-47f4-8425-67443d3098ac" path="/var/lib/kubelet/pods/90a1aef0-b76b-47f4-8425-67443d3098ac/volumes"
Jul 23 18:55:53.736007 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.735966    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"]
Jul 23 18:55:53.736688 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.736667    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8100815e-80a2-42ab-a0f8-da8c0eb77af5" containerName="maas-api"
Jul 23 18:55:53.736786 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.736691    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8100815e-80a2-42ab-a0f8-da8c0eb77af5" containerName="maas-api"
Jul 23 18:55:53.736786 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.736745    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a1aef0-b76b-47f4-8425-67443d3098ac" containerName="revoke-keys"
Jul 23 18:55:53.736786 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.736754    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a1aef0-b76b-47f4-8425-67443d3098ac" containerName="revoke-keys"
Jul 23 18:55:53.736883 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.736837    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8100815e-80a2-42ab-a0f8-da8c0eb77af5" containerName="maas-api"
Jul 23 18:55:53.736883 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.736850    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a1aef0-b76b-47f4-8425-67443d3098ac" containerName="revoke-keys"
Jul 23 18:55:53.740262 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.740239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:53.743646 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.743620    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-3f12e9-serving-cert\""
Jul 23 18:55:53.743775 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.743741    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 18:55:53.744493 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.744476    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jvms6\""
Jul 23 18:55:53.746868 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.746114    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 18:55:53.749718 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.749697    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"]
Jul 23 18:55:53.859973 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.859941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c28355-c254-4a07-ba14-d3b578c1b238-maas-api-tls\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:53.860156 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.859986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c28355-c254-4a07-ba14-d3b578c1b238-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:53.860156 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.860008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mfv7\" (UniqueName: \"kubernetes.io/projected/36c28355-c254-4a07-ba14-d3b578c1b238-kube-api-access-6mfv7\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:53.960650 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.960613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c28355-c254-4a07-ba14-d3b578c1b238-maas-api-tls\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:53.960849 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.960663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c28355-c254-4a07-ba14-d3b578c1b238-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:53.960849 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.960679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mfv7\" (UniqueName: \"kubernetes.io/projected/36c28355-c254-4a07-ba14-d3b578c1b238-kube-api-access-6mfv7\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:53.960849 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:55:53.960792    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-serving-cert: secret "maas-api-e2e-shared-a-3f12e9-serving-cert" not found
Jul 23 18:55:53.961027 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:55:53.960882    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36c28355-c254-4a07-ba14-d3b578c1b238-maas-api-tls podName:36c28355-c254-4a07-ba14-d3b578c1b238 nodeName:}" failed. No retries permitted until 2026-07-23 18:55:54.460861475 +0000 UTC m=+2350.308551980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36c28355-c254-4a07-ba14-d3b578c1b238-maas-api-tls") pod "maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc" (UID: "36c28355-c254-4a07-ba14-d3b578c1b238") : secret "maas-api-e2e-shared-a-3f12e9-serving-cert" not found
Jul 23 18:55:53.961335 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.961317    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c28355-c254-4a07-ba14-d3b578c1b238-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:53.973152 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:53.973125    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mfv7\" (UniqueName: \"kubernetes.io/projected/36c28355-c254-4a07-ba14-d3b578c1b238-kube-api-access-6mfv7\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:54.464776 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:54.464733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c28355-c254-4a07-ba14-d3b578c1b238-maas-api-tls\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:54.467089 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:54.467062    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c28355-c254-4a07-ba14-d3b578c1b238-maas-api-tls\") pod \"maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:54.651984 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:54.651950    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:54.809520 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:54.809488    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"]
Jul 23 18:55:54.812625 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:55:54.812596    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c28355_c254_4a07_ba14_d3b578c1b238.slice/crio-69c0086d52048474d49eb053ae863f45e4a0fe0c829dc5e741e1cc961a6fe808 WatchSource:0}: Error finding container 69c0086d52048474d49eb053ae863f45e4a0fe0c829dc5e741e1cc961a6fe808: Status 404 returned error can't find the container with id 69c0086d52048474d49eb053ae863f45e4a0fe0c829dc5e741e1cc961a6fe808
Jul 23 18:55:55.174848 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:55.174755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc" event={"ID":"36c28355-c254-4a07-ba14-d3b578c1b238","Type":"ContainerStarted","Data":"69c0086d52048474d49eb053ae863f45e4a0fe0c829dc5e741e1cc961a6fe808"}
Jul 23 18:55:56.182035 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:56.181996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc" event={"ID":"36c28355-c254-4a07-ba14-d3b578c1b238","Type":"ContainerStarted","Data":"69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52"}
Jul 23 18:55:56.182530 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:56.182061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:55:56.202519 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:56.202473    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc" podStartSLOduration=2.738584786 podStartE2EDuration="3.202459343s" totalImagesPullingTime="463.874557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:55:56.200708589 +0000 UTC m=+2352.048399113" watchObservedRunningTime="2026-07-23 18:55:56.202459343 +0000 UTC m=+2352.050149869"
Jul 23 18:55:58.688509 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:55:58.688471    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:56:02.014433 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.014396    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"]
Jul 23 18:56:02.018423 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.018397    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.021404 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.021380    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-60de5a-serving-cert\""
Jul 23 18:56:02.038666 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.038631    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"]
Jul 23 18:56:02.132244 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.132204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrtf6\" (UniqueName: \"kubernetes.io/projected/312ce815-1a40-4a2a-97fb-1dde041829db-kube-api-access-lrtf6\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.132455 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.132349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312ce815-1a40-4a2a-97fb-1dde041829db-maas-api-tls\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.132455 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.132377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ce815-1a40-4a2a-97fb-1dde041829db-openshift-service-ca\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.193754 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.193726    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 18:56:02.233587 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.233554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrtf6\" (UniqueName: \"kubernetes.io/projected/312ce815-1a40-4a2a-97fb-1dde041829db-kube-api-access-lrtf6\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.233795 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.233617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312ce815-1a40-4a2a-97fb-1dde041829db-maas-api-tls\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.233795 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.233645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ce815-1a40-4a2a-97fb-1dde041829db-openshift-service-ca\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.233795 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:56:02.233715    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-serving-cert: secret "maas-api-e2e-shared-b-60de5a-serving-cert" not found
Jul 23 18:56:02.233795 ip-10-0-140-91 kubenswrapper[2561]: E0723 18:56:02.233782    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/312ce815-1a40-4a2a-97fb-1dde041829db-maas-api-tls podName:312ce815-1a40-4a2a-97fb-1dde041829db nodeName:}" failed. No retries permitted until 2026-07-23 18:56:02.733760086 +0000 UTC m=+2358.581450595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/312ce815-1a40-4a2a-97fb-1dde041829db-maas-api-tls") pod "maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx" (UID: "312ce815-1a40-4a2a-97fb-1dde041829db") : secret "maas-api-e2e-shared-b-60de5a-serving-cert" not found
Jul 23 18:56:02.234317 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.234277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ce815-1a40-4a2a-97fb-1dde041829db-openshift-service-ca\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.268523 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.268449    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrtf6\" (UniqueName: \"kubernetes.io/projected/312ce815-1a40-4a2a-97fb-1dde041829db-kube-api-access-lrtf6\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.738368 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.738326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312ce815-1a40-4a2a-97fb-1dde041829db-maas-api-tls\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.740782 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.740756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312ce815-1a40-4a2a-97fb-1dde041829db-maas-api-tls\") pod \"maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:02.930034 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:02.930003    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:03.087475 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:03.087447    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"]
Jul 23 18:56:03.220802 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:03.220763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx" event={"ID":"312ce815-1a40-4a2a-97fb-1dde041829db","Type":"ContainerStarted","Data":"e8e89d7dafe1f84a332a009bf30fdf40492e6f31e8941943dec929bdabfae234"}
Jul 23 18:56:04.228165 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:04.228127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx" event={"ID":"312ce815-1a40-4a2a-97fb-1dde041829db","Type":"ContainerStarted","Data":"c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08"}
Jul 23 18:56:04.228581 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:04.228324    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:04.248016 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:04.247968    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx" podStartSLOduration=2.818261571 podStartE2EDuration="3.247953154s" totalImagesPullingTime="429.691583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:56:04.24605102 +0000 UTC m=+2360.093741546" watchObservedRunningTime="2026-07-23 18:56:04.247953154 +0000 UTC m=+2360.095643724"
Jul 23 18:56:10.239686 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:10.239659    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 18:56:15.293373 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:15.293339    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:56:32.586672 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:32.586634    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:56:44.792464 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:44.792436    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:56:44.800128 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:44.800109    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 18:56:44.939546 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:56:44.939518    2561 scope.go:117] "RemoveContainer" containerID="df1c902905fa4f1e6f8fe4a6027141a6e92528920923e1beeb592f08f6bf8a32"
Jul 23 18:57:04.590316 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:57:04.590267    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:57:19.201201 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:57:19.201161    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:57:33.881505 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:57:33.881466    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:57:51.188978 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:57:51.188940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:58:07.590579 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:58:07.590543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:58:26.588237 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:58:26.588196    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:58:44.278892 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:58:44.278814    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:59:02.490780 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:02.490738    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:59:13.604838 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.604803    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"]
Jul 23 18:59:13.608488 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.608470    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.617600 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.617571    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f12e9\"/\"sub-test-model-3f12e9-kserve-self-signed-certs\""
Jul 23 18:59:13.618060 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.618038    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f12e9\"/\"default-dockercfg-56pzq\""
Jul 23 18:59:13.619231 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.618830    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3f12e9\"/\"kube-root-ca.crt\""
Jul 23 18:59:13.619231 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.618941    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3f12e9\"/\"openshift-service-ca.crt\""
Jul 23 18:59:13.619231 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.619168    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"]
Jul 23 18:59:13.779618 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.779584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tls-certs\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.779829 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.779633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-dshm\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.779829 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.779801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-model-cache\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.779961 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.779845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tmp-dir\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.779961 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.779898    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mp2k\" (UniqueName: \"kubernetes.io/projected/0b6c02ef-6c91-4bea-9a15-99bdfe109961-kube-api-access-9mp2k\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.779961 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.779939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-home\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.880583 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.880495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-model-cache\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.880583 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.880533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tmp-dir\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.880583 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.880555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mp2k\" (UniqueName: \"kubernetes.io/projected/0b6c02ef-6c91-4bea-9a15-99bdfe109961-kube-api-access-9mp2k\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.880867 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.880594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-home\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.880867 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.880621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tls-certs\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.880867 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.880653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-dshm\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.881017 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.880881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-model-cache\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.881017 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.880933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tmp-dir\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.881097 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.881071    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-home\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.882859 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.882836    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-dshm\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.883062 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.883046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tls-certs\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.889092 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.889067    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mp2k\" (UniqueName: \"kubernetes.io/projected/0b6c02ef-6c91-4bea-9a15-99bdfe109961-kube-api-access-9mp2k\") pod \"sub-test-model-3f12e9-kserve-65448fcf47-dfbtm\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") " pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:13.928127 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:13.928092    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:14.059773 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:14.059723    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"]
Jul 23 18:59:14.062229 ip-10-0-140-91 kubenswrapper[2561]: W0723 18:59:14.062205    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b6c02ef_6c91_4bea_9a15_99bdfe109961.slice/crio-b0c33352cd40d629bbff47e258611b69dc81a292a6709827f547a7b7eb891d1b WatchSource:0}: Error finding container b0c33352cd40d629bbff47e258611b69dc81a292a6709827f547a7b7eb891d1b: Status 404 returned error can't find the container with id b0c33352cd40d629bbff47e258611b69dc81a292a6709827f547a7b7eb891d1b
Jul 23 18:59:14.178242 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:14.178155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm" event={"ID":"0b6c02ef-6c91-4bea-9a15-99bdfe109961","Type":"ContainerStarted","Data":"b0c33352cd40d629bbff47e258611b69dc81a292a6709827f547a7b7eb891d1b"}
Jul 23 18:59:15.185208 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:15.185177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm" event={"ID":"0b6c02ef-6c91-4bea-9a15-99bdfe109961","Type":"ContainerStarted","Data":"dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6"}
Jul 23 18:59:15.214296 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:15.214234    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm" podStartSLOduration=1.175026884 podStartE2EDuration="2.214219216s" totalImagesPullingTime="1.039192332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:59:15.211633481 +0000 UTC m=+2551.059324008" watchObservedRunningTime="2026-07-23 18:59:15.214219216 +0000 UTC m=+2551.061909722"
Jul 23 18:59:16.282305 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:16.282255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 18:59:23.928619 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:23.928578    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:23.929030 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:23.928825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:23.941299 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:23.941256    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:24.241499 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:24.241472    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 18:59:45.788688 ip-10-0-140-91 kubenswrapper[2561]: I0723 18:59:45.788650    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:00:00.137203 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.137167    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs"]
Jul 23 19:00:00.141071 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.141043    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2"]
Jul 23 19:00:00.141219 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.141154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs"
Jul 23 19:00:00.143726 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.143705    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 19:00:00.144863 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.144843    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr"]
Jul 23 19:00:00.144965 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.144943    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2"
Jul 23 19:00:00.148313 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.148271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr"
Jul 23 19:00:00.149985 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.149966    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2"]
Jul 23 19:00:00.154918 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.154887    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs"]
Jul 23 19:00:00.156080 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.156056    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs"
Jul 23 19:00:00.157828 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.157807    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr"]
Jul 23 19:00:00.164539 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.164521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2"
Jul 23 19:00:00.192081 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.192051    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr"
Jul 23 19:00:00.326641 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.326557    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs"]
Jul 23 19:00:00.328962 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:00:00.328932    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b03216_3eff_4435_975c_c918ae179d84.slice/crio-8c45769d814b92a8b86f0653ca6e391e488476d3f784669a73d0835bcd20e24f WatchSource:0}: Error finding container 8c45769d814b92a8b86f0653ca6e391e488476d3f784669a73d0835bcd20e24f: Status 404 returned error can't find the container with id 8c45769d814b92a8b86f0653ca6e391e488476d3f784669a73d0835bcd20e24f
Jul 23 19:00:00.351136 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.351110    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2"]
Jul 23 19:00:00.360334 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:00:00.360279    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a8529c_4c90_4d65_9ead_4f5e9e253f93.slice/crio-b644efea4b9f09598da5a1735619a6dc714d33184bb3e001984fb110f3022400 WatchSource:0}: Error finding container b644efea4b9f09598da5a1735619a6dc714d33184bb3e001984fb110f3022400: Status 404 returned error can't find the container with id b644efea4b9f09598da5a1735619a6dc714d33184bb3e001984fb110f3022400
Jul 23 19:00:00.378308 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.378253    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr"]
Jul 23 19:00:00.382247 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:00:00.382207    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea185c61_69ce_4be2_8f93_20a4c5909d86.slice/crio-aae92273cdd4cee7f0844f728f17f35bfa2aaac60f55abb170095b42b08966dc WatchSource:0}: Error finding container aae92273cdd4cee7f0844f728f17f35bfa2aaac60f55abb170095b42b08966dc: Status 404 returned error can't find the container with id aae92273cdd4cee7f0844f728f17f35bfa2aaac60f55abb170095b42b08966dc
Jul 23 19:00:00.414461 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.414431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr" event={"ID":"ea185c61-69ce-4be2-8f93-20a4c5909d86","Type":"ContainerStarted","Data":"aae92273cdd4cee7f0844f728f17f35bfa2aaac60f55abb170095b42b08966dc"}
Jul 23 19:00:00.415551 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.415528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs" event={"ID":"f1b03216-3eff-4435-975c-c918ae179d84","Type":"ContainerStarted","Data":"8c45769d814b92a8b86f0653ca6e391e488476d3f784669a73d0835bcd20e24f"}
Jul 23 19:00:00.416703 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:00.416684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2" event={"ID":"a9a8529c-4c90-4d65-9ead-4f5e9e253f93","Type":"ContainerStarted","Data":"b644efea4b9f09598da5a1735619a6dc714d33184bb3e001984fb110f3022400"}
Jul 23 19:00:01.423529 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:01.423496    2561 generic.go:356] "Generic (PLEG): container finished" podID="f1b03216-3eff-4435-975c-c918ae179d84" containerID="ae3c67ae3ea39a5400ecb40bb7e08fcb2280696500472a663766d1bfa7fa2545" exitCode=0
Jul 23 19:00:01.423926 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:01.423576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs" event={"ID":"f1b03216-3eff-4435-975c-c918ae179d84","Type":"ContainerDied","Data":"ae3c67ae3ea39a5400ecb40bb7e08fcb2280696500472a663766d1bfa7fa2545"}
Jul 23 19:00:01.425428 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:01.425396    2561 generic.go:356] "Generic (PLEG): container finished" podID="a9a8529c-4c90-4d65-9ead-4f5e9e253f93" containerID="eae3ed445eeb3bbe24bc4b275c242151cc4fe7821c78fc94e17dc5cea8db0d1c" exitCode=0
Jul 23 19:00:01.425855 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:01.425494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2" event={"ID":"a9a8529c-4c90-4d65-9ead-4f5e9e253f93","Type":"ContainerDied","Data":"eae3ed445eeb3bbe24bc4b275c242151cc4fe7821c78fc94e17dc5cea8db0d1c"}
Jul 23 19:00:01.427266 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:01.427244    2561 generic.go:356] "Generic (PLEG): container finished" podID="ea185c61-69ce-4be2-8f93-20a4c5909d86" containerID="c18fe810df604b7cec4119805798b98456c2e26156ff66a10918fb9a7845c20a" exitCode=0
Jul 23 19:00:01.427397 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:01.427306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr" event={"ID":"ea185c61-69ce-4be2-8f93-20a4c5909d86","Type":"ContainerDied","Data":"c18fe810df604b7cec4119805798b98456c2e26156ff66a10918fb9a7845c20a"}
Jul 23 19:00:02.568729 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:02.568700    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr"
Jul 23 19:00:02.622802 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:02.622779    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs"
Jul 23 19:00:02.625683 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:02.625661    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2"
Jul 23 19:00:03.439139 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:03.439113    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs"
Jul 23 19:00:03.439139 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:03.439138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747220-tjtgs" event={"ID":"f1b03216-3eff-4435-975c-c918ae179d84","Type":"ContainerDied","Data":"8c45769d814b92a8b86f0653ca6e391e488476d3f784669a73d0835bcd20e24f"}
Jul 23 19:00:03.439400 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:03.439184    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c45769d814b92a8b86f0653ca6e391e488476d3f784669a73d0835bcd20e24f"
Jul 23 19:00:03.440476 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:03.440457    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2"
Jul 23 19:00:03.440621 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:03.440492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2" event={"ID":"a9a8529c-4c90-4d65-9ead-4f5e9e253f93","Type":"ContainerDied","Data":"b644efea4b9f09598da5a1735619a6dc714d33184bb3e001984fb110f3022400"}
Jul 23 19:00:03.440621 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:03.440536    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b644efea4b9f09598da5a1735619a6dc714d33184bb3e001984fb110f3022400"
Jul 23 19:00:03.441917 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:03.441895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr" event={"ID":"ea185c61-69ce-4be2-8f93-20a4c5909d86","Type":"ContainerDied","Data":"aae92273cdd4cee7f0844f728f17f35bfa2aaac60f55abb170095b42b08966dc"}
Jul 23 19:00:03.441999 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:03.441946    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aae92273cdd4cee7f0844f728f17f35bfa2aaac60f55abb170095b42b08966dc"
Jul 23 19:00:03.441999 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:03.441905    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr"
Jul 23 19:00:12.088926 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:12.088884    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:00:29.354682 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:29.354649    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"]
Jul 23 19:00:33.696401 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.696368    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"]
Jul 23 19:00:33.697503 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.697473    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1b03216-3eff-4435-975c-c918ae179d84" containerName="cleanup"
Jul 23 19:00:33.697503 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.697505    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1b03216-3eff-4435-975c-c918ae179d84" containerName="cleanup"
Jul 23 19:00:33.697654 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.697525    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea185c61-69ce-4be2-8f93-20a4c5909d86" containerName="cleanup"
Jul 23 19:00:33.697654 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.697535    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea185c61-69ce-4be2-8f93-20a4c5909d86" containerName="cleanup"
Jul 23 19:00:33.697654 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.697578    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a8529c-4c90-4d65-9ead-4f5e9e253f93" containerName="cleanup"
Jul 23 19:00:33.697654 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.697587    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a8529c-4c90-4d65-9ead-4f5e9e253f93" containerName="cleanup"
Jul 23 19:00:33.697784 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.697683    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9a8529c-4c90-4d65-9ead-4f5e9e253f93" containerName="cleanup"
Jul 23 19:00:33.697784 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.697700    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1b03216-3eff-4435-975c-c918ae179d84" containerName="cleanup"
Jul 23 19:00:33.697784 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.697710    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea185c61-69ce-4be2-8f93-20a4c5909d86" containerName="cleanup"
Jul 23 19:00:33.701549 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.701523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.704201 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.704180    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f12e9\"/\"rate-test-model-3f12e9-kserve-self-signed-certs\""
Jul 23 19:00:33.709913 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.709887    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"]
Jul 23 19:00:33.797879 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.797843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-model-cache\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.798049 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.797902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-dshm\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.798049 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.797942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c846ef-1765-4a1a-a137-cb08f4385ccf-tls-certs\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.798049 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.797963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-home\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.798049 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.797983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-tmp-dir\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.798049 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.798012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7tp\" (UniqueName: \"kubernetes.io/projected/77c846ef-1765-4a1a-a137-cb08f4385ccf-kube-api-access-rc7tp\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.899210 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.899175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c846ef-1765-4a1a-a137-cb08f4385ccf-tls-certs\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.899210 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.899213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-home\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.899460 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.899234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-tmp-dir\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.899460 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.899256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7tp\" (UniqueName: \"kubernetes.io/projected/77c846ef-1765-4a1a-a137-cb08f4385ccf-kube-api-access-rc7tp\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.899460 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.899329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-model-cache\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.899460 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.899375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-dshm\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.899729 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.899703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-tmp-dir\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.899838 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.899733    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-home\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.899838 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.899769    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-model-cache\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.901619 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.901602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-dshm\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.901905 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.901885    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c846ef-1765-4a1a-a137-cb08f4385ccf-tls-certs\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:33.908009 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:33.907986    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7tp\" (UniqueName: \"kubernetes.io/projected/77c846ef-1765-4a1a-a137-cb08f4385ccf-kube-api-access-rc7tp\") pod \"rate-test-model-3f12e9-kserve-b77598b99-qsdxj\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") " pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:34.015182 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:34.015148    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:34.149516 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:34.149494    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"]
Jul 23 19:00:34.151893 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:00:34.151859    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77c846ef_1765_4a1a_a137_cb08f4385ccf.slice/crio-3e826abb40aaf4ccc11fc188c996a574eb4c9299bde99e21720d79df60a189e4 WatchSource:0}: Error finding container 3e826abb40aaf4ccc11fc188c996a574eb4c9299bde99e21720d79df60a189e4: Status 404 returned error can't find the container with id 3e826abb40aaf4ccc11fc188c996a574eb4c9299bde99e21720d79df60a189e4
Jul 23 19:00:34.599928 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:34.599885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj" event={"ID":"77c846ef-1765-4a1a-a137-cb08f4385ccf","Type":"ContainerStarted","Data":"73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41"}
Jul 23 19:00:34.599928 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:34.599923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj" event={"ID":"77c846ef-1765-4a1a-a137-cb08f4385ccf","Type":"ContainerStarted","Data":"3e826abb40aaf4ccc11fc188c996a574eb4c9299bde99e21720d79df60a189e4"}
Jul 23 19:00:34.621569 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:34.621513    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj" podStartSLOduration=1.621492403 podStartE2EDuration="1.621492403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:00:34.620598912 +0000 UTC m=+2630.468289440" watchObservedRunningTime="2026-07-23 19:00:34.621492403 +0000 UTC m=+2630.469182930"
Jul 23 19:00:34.789833 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:34.789801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:00:44.016035 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.015991    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:44.016604 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.016048    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:44.028488 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.028467    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:44.374593 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.374484    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm" podUID="0b6c02ef-6c91-4bea-9a15-99bdfe109961" containerName="main" containerID="cri-o://dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6" gracePeriod=45
Jul 23 19:00:44.630173 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.630111    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 19:00:44.657368 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.657337    2561 generic.go:356] "Generic (PLEG): container finished" podID="0b6c02ef-6c91-4bea-9a15-99bdfe109961" containerID="dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6" exitCode=0
Jul 23 19:00:44.657496 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.657374    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"
Jul 23 19:00:44.657496 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.657424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm" event={"ID":"0b6c02ef-6c91-4bea-9a15-99bdfe109961","Type":"ContainerDied","Data":"dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6"}
Jul 23 19:00:44.657496 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.657471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm" event={"ID":"0b6c02ef-6c91-4bea-9a15-99bdfe109961","Type":"ContainerDied","Data":"b0c33352cd40d629bbff47e258611b69dc81a292a6709827f547a7b7eb891d1b"}
Jul 23 19:00:44.657610 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.657496    2561 scope.go:117] "RemoveContainer" containerID="dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6"
Jul 23 19:00:44.668036 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.668016    2561 scope.go:117] "RemoveContainer" containerID="dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6"
Jul 23 19:00:44.668381 ip-10-0-140-91 kubenswrapper[2561]: E0723 19:00:44.668357    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6\": container with ID starting with dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6 not found: ID does not exist" containerID="dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6"
Jul 23 19:00:44.668459 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.668394    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6"} err="failed to get container status \"dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6\": rpc error: code = NotFound desc = could not find container \"dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6\": container with ID starting with dd545a422c5b9175d5a4c3e0b3bf04941a66363045494933d6079b7b46f31cd6 not found: ID does not exist"
Jul 23 19:00:44.671582 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.671558    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:00:44.700173 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.700143    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:00:44.805714 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.805673    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-model-cache\") pod \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") "
Jul 23 19:00:44.805886 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.805769    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mp2k\" (UniqueName: \"kubernetes.io/projected/0b6c02ef-6c91-4bea-9a15-99bdfe109961-kube-api-access-9mp2k\") pod \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") "
Jul 23 19:00:44.805886 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.805844    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tmp-dir\") pod \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") "
Jul 23 19:00:44.805886 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.805871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tls-certs\") pod \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") "
Jul 23 19:00:44.806056 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.805910    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-dshm\") pod \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") "
Jul 23 19:00:44.806056 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.805949    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-home\") pod \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\" (UID: \"0b6c02ef-6c91-4bea-9a15-99bdfe109961\") "
Jul 23 19:00:44.806056 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.805961    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b6c02ef-6c91-4bea-9a15-99bdfe109961" (UID: "0b6c02ef-6c91-4bea-9a15-99bdfe109961"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:00:44.806214 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.806108    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b6c02ef-6c91-4bea-9a15-99bdfe109961" (UID: "0b6c02ef-6c91-4bea-9a15-99bdfe109961"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:00:44.806312 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.806271    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-model-cache\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:00:44.806312 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.806271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-home" (OuterVolumeSpecName: "home") pod "0b6c02ef-6c91-4bea-9a15-99bdfe109961" (UID: "0b6c02ef-6c91-4bea-9a15-99bdfe109961"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:00:44.806435 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.806306    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tmp-dir\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:00:44.808050 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.808028    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-dshm" (OuterVolumeSpecName: "dshm") pod "0b6c02ef-6c91-4bea-9a15-99bdfe109961" (UID: "0b6c02ef-6c91-4bea-9a15-99bdfe109961"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:00:44.808333 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.808310    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b6c02ef-6c91-4bea-9a15-99bdfe109961" (UID: "0b6c02ef-6c91-4bea-9a15-99bdfe109961"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 19:00:44.808333 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.808321    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6c02ef-6c91-4bea-9a15-99bdfe109961-kube-api-access-9mp2k" (OuterVolumeSpecName: "kube-api-access-9mp2k") pod "0b6c02ef-6c91-4bea-9a15-99bdfe109961" (UID: "0b6c02ef-6c91-4bea-9a15-99bdfe109961"). InnerVolumeSpecName "kube-api-access-9mp2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:00:44.907339 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.907222    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mp2k\" (UniqueName: \"kubernetes.io/projected/0b6c02ef-6c91-4bea-9a15-99bdfe109961-kube-api-access-9mp2k\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:00:44.907339 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.907254    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6c02ef-6c91-4bea-9a15-99bdfe109961-tls-certs\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:00:44.907339 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.907268    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-dshm\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:00:44.907339 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.907279    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6c02ef-6c91-4bea-9a15-99bdfe109961-home\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:00:44.980972 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.980942    2561 scope.go:117] "RemoveContainer" containerID="a0e647e9452efcd2c19f553b6f7996316e4c33fb73109a8eac3c1ead1790a2b4"
Jul 23 19:00:44.981774 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.981753    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"]
Jul 23 19:00:44.986489 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.986458    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/sub-test-model-3f12e9-kserve-65448fcf47-dfbtm"]
Jul 23 19:00:44.992330 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:44.992310    2561 scope.go:117] "RemoveContainer" containerID="bbdffae42be618d881611eeed33a3c47fb7ff72fa989e5716e23deed7f2296d4"
Jul 23 19:00:46.761083 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:00:46.761050    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6c02ef-6c91-4bea-9a15-99bdfe109961" path="/var/lib/kubelet/pods/0b6c02ef-6c91-4bea-9a15-99bdfe109961/volumes"
Jul 23 19:01:01.417341 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.417305    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"]
Jul 23 19:01:01.417756 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.417714    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6c02ef-6c91-4bea-9a15-99bdfe109961" containerName="main"
Jul 23 19:01:01.417756 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.417728    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6c02ef-6c91-4bea-9a15-99bdfe109961" containerName="main"
Jul 23 19:01:01.417833 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.417811    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6c02ef-6c91-4bea-9a15-99bdfe109961" containerName="main"
Jul 23 19:01:01.420977 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.420959    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.423580 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.423554    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"rate-test-model-60de5a-kserve-self-signed-certs\""
Jul 23 19:01:01.424577 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.424559    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"default-dockercfg-2lczk\""
Jul 23 19:01:01.424721 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.424670    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"openshift-service-ca.crt\""
Jul 23 19:01:01.424721 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.424711    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"kube-root-ca.crt\""
Jul 23 19:01:01.434508 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.434486    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"]
Jul 23 19:01:01.558157 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.558123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv57k\" (UniqueName: \"kubernetes.io/projected/5fb16b2f-76c2-4886-9686-ad38d1821981-kube-api-access-wv57k\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.558157 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.558162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-model-cache\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.558394 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.558191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-home\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.558394 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.558235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-dshm\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.558394 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.558307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-tmp-dir\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.558488 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.558397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb16b2f-76c2-4886-9686-ad38d1821981-tls-certs\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.659385 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.659351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv57k\" (UniqueName: \"kubernetes.io/projected/5fb16b2f-76c2-4886-9686-ad38d1821981-kube-api-access-wv57k\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.659548 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.659395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-model-cache\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.659548 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.659515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-home\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.659548 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.659544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-dshm\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.659662 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.659559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-tmp-dir\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.659662 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.659630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb16b2f-76c2-4886-9686-ad38d1821981-tls-certs\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.659789 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.659771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-model-cache\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.659827 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.659807    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-home\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.659962 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.659941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-tmp-dir\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.661815 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.661793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-dshm\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.662084 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.662065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb16b2f-76c2-4886-9686-ad38d1821981-tls-certs\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.668696 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.668637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv57k\" (UniqueName: \"kubernetes.io/projected/5fb16b2f-76c2-4886-9686-ad38d1821981-kube-api-access-wv57k\") pod \"rate-test-model-60de5a-kserve-55f664484c-m4dpf\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") " pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.730348 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.730323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:01.874948 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:01.874921    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"]
Jul 23 19:01:01.877500 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:01:01.877476    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fb16b2f_76c2_4886_9686_ad38d1821981.slice/crio-a5a700478ccf6cc1739cf3383822bc3ce47e5287c5c6d0932047d72ed1754695 WatchSource:0}: Error finding container a5a700478ccf6cc1739cf3383822bc3ce47e5287c5c6d0932047d72ed1754695: Status 404 returned error can't find the container with id a5a700478ccf6cc1739cf3383822bc3ce47e5287c5c6d0932047d72ed1754695
Jul 23 19:01:02.761607 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:02.761570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf" event={"ID":"5fb16b2f-76c2-4886-9686-ad38d1821981","Type":"ContainerStarted","Data":"58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03"}
Jul 23 19:01:02.761994 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:02.761616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf" event={"ID":"5fb16b2f-76c2-4886-9686-ad38d1821981","Type":"ContainerStarted","Data":"a5a700478ccf6cc1739cf3383822bc3ce47e5287c5c6d0932047d72ed1754695"}
Jul 23 19:01:02.786784 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:02.786728    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf" podStartSLOduration=1.786712262 podStartE2EDuration="1.786712262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:01:02.785587241 +0000 UTC m=+2658.633277769" watchObservedRunningTime="2026-07-23 19:01:02.786712262 +0000 UTC m=+2658.634402789"
Jul 23 19:01:05.088123 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:05.088070    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:01:11.731350 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:11.731314    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:11.731832 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:11.731370    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:11.743994 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:11.743968    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:11.818933 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:11.818900    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:16.106595 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:16.106560    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:01:33.786801 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:33.786761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:01:37.919413 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:37.919319    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"]
Jul 23 19:01:38.689355 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:38.689311    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"]
Jul 23 19:01:39.639437 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:39.639396    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7434-140634787859376"]
Jul 23 19:01:39.643038 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:39.643022    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787859376"
Jul 23 19:01:39.645541 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:39.645518    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-92csk\""
Jul 23 19:01:39.651993 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:39.651971    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787859376"]
Jul 23 19:01:39.721614 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:39.721582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklj7\" (UniqueName: \"kubernetes.io/projected/84309dbb-c11c-4695-88e8-d15fee02f598-kube-api-access-tklj7\") pod \"test-curl-7434-140634787859376\" (UID: \"84309dbb-c11c-4695-88e8-d15fee02f598\") " pod="opendatahub/test-curl-7434-140634787859376"
Jul 23 19:01:39.822948 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:39.822915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklj7\" (UniqueName: \"kubernetes.io/projected/84309dbb-c11c-4695-88e8-d15fee02f598-kube-api-access-tklj7\") pod \"test-curl-7434-140634787859376\" (UID: \"84309dbb-c11c-4695-88e8-d15fee02f598\") " pod="opendatahub/test-curl-7434-140634787859376"
Jul 23 19:01:39.832360 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:39.832331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklj7\" (UniqueName: \"kubernetes.io/projected/84309dbb-c11c-4695-88e8-d15fee02f598-kube-api-access-tklj7\") pod \"test-curl-7434-140634787859376\" (UID: \"84309dbb-c11c-4695-88e8-d15fee02f598\") " pod="opendatahub/test-curl-7434-140634787859376"
Jul 23 19:01:39.953332 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:39.953308    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787859376"
Jul 23 19:01:40.086978 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:40.086957    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787859376"]
Jul 23 19:01:40.089013 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:01:40.088976    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84309dbb_c11c_4695_88e8_d15fee02f598.slice/crio-2082bf3353f2fae00a2989a468575803c13f7cb9dbc1c9469e449bda3aebe719 WatchSource:0}: Error finding container 2082bf3353f2fae00a2989a468575803c13f7cb9dbc1c9469e449bda3aebe719: Status 404 returned error can't find the container with id 2082bf3353f2fae00a2989a468575803c13f7cb9dbc1c9469e449bda3aebe719
Jul 23 19:01:40.090473 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:40.090452    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 19:01:40.953042 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:40.953006    2561 generic.go:356] "Generic (PLEG): container finished" podID="84309dbb-c11c-4695-88e8-d15fee02f598" containerID="36e0c780f68897b6a21631fc6843c690eff2fb25e0c76f192d7eff9e40f259ab" exitCode=0
Jul 23 19:01:40.953451 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:40.953094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787859376" event={"ID":"84309dbb-c11c-4695-88e8-d15fee02f598","Type":"ContainerDied","Data":"36e0c780f68897b6a21631fc6843c690eff2fb25e0c76f192d7eff9e40f259ab"}
Jul 23 19:01:40.953451 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:40.953141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787859376" event={"ID":"84309dbb-c11c-4695-88e8-d15fee02f598","Type":"ContainerStarted","Data":"2082bf3353f2fae00a2989a468575803c13f7cb9dbc1c9469e449bda3aebe719"}
Jul 23 19:01:42.097595 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.097573    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787859376"
Jul 23 19:01:42.138049 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.138005    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7434-140634787859376_84309dbb-c11c-4695-88e8-d15fee02f598/test-curl-7434-140634787859376/0.log"
Jul 23 19:01:42.141219 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.141194    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tklj7\" (UniqueName: \"kubernetes.io/projected/84309dbb-c11c-4695-88e8-d15fee02f598-kube-api-access-tklj7\") pod \"84309dbb-c11c-4695-88e8-d15fee02f598\" (UID: \"84309dbb-c11c-4695-88e8-d15fee02f598\") "
Jul 23 19:01:42.143319 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.143277    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84309dbb-c11c-4695-88e8-d15fee02f598-kube-api-access-tklj7" (OuterVolumeSpecName: "kube-api-access-tklj7") pod "84309dbb-c11c-4695-88e8-d15fee02f598" (UID: "84309dbb-c11c-4695-88e8-d15fee02f598"). InnerVolumeSpecName "kube-api-access-tklj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:01:42.165011 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.164980    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7434-140634787859376"]
Jul 23 19:01:42.169000 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.168978    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7434-140634787859376"]
Jul 23 19:01:42.242570 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.242488    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tklj7\" (UniqueName: \"kubernetes.io/projected/84309dbb-c11c-4695-88e8-d15fee02f598-kube-api-access-tklj7\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:42.296557 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.296523    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7434-140634787858992"]
Jul 23 19:01:42.296961 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.296947    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84309dbb-c11c-4695-88e8-d15fee02f598" containerName="test-curl-7434-140634787859376"
Jul 23 19:01:42.297026 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.296963    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="84309dbb-c11c-4695-88e8-d15fee02f598" containerName="test-curl-7434-140634787859376"
Jul 23 19:01:42.297069 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.297026    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="84309dbb-c11c-4695-88e8-d15fee02f598" containerName="test-curl-7434-140634787859376"
Jul 23 19:01:42.299973 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.299957    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787858992"
Jul 23 19:01:42.308328 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.308305    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787858992"]
Jul 23 19:01:42.343684 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.343654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhbw\" (UniqueName: \"kubernetes.io/projected/852b0856-087c-4326-82e1-6e4aaea94ec8-kube-api-access-shhbw\") pod \"test-curl-7434-140634787858992\" (UID: \"852b0856-087c-4326-82e1-6e4aaea94ec8\") " pod="opendatahub/test-curl-7434-140634787858992"
Jul 23 19:01:42.444623 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.444590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhbw\" (UniqueName: \"kubernetes.io/projected/852b0856-087c-4326-82e1-6e4aaea94ec8-kube-api-access-shhbw\") pod \"test-curl-7434-140634787858992\" (UID: \"852b0856-087c-4326-82e1-6e4aaea94ec8\") " pod="opendatahub/test-curl-7434-140634787858992"
Jul 23 19:01:42.453967 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.453935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhbw\" (UniqueName: \"kubernetes.io/projected/852b0856-087c-4326-82e1-6e4aaea94ec8-kube-api-access-shhbw\") pod \"test-curl-7434-140634787858992\" (UID: \"852b0856-087c-4326-82e1-6e4aaea94ec8\") " pod="opendatahub/test-curl-7434-140634787858992"
Jul 23 19:01:42.610116 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.610032    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787858992"
Jul 23 19:01:42.741542 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.741500    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787858992"]
Jul 23 19:01:42.748330 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:01:42.748264    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod852b0856_087c_4326_82e1_6e4aaea94ec8.slice/crio-825983a9d6bf351b5198b748b8b195688a3c3561f3c6b90cc5af4cf44cf7db67 WatchSource:0}: Error finding container 825983a9d6bf351b5198b748b8b195688a3c3561f3c6b90cc5af4cf44cf7db67: Status 404 returned error can't find the container with id 825983a9d6bf351b5198b748b8b195688a3c3561f3c6b90cc5af4cf44cf7db67
Jul 23 19:01:42.760656 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.760627    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84309dbb-c11c-4695-88e8-d15fee02f598" path="/var/lib/kubelet/pods/84309dbb-c11c-4695-88e8-d15fee02f598/volumes"
Jul 23 19:01:42.964545 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.964508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787858992" event={"ID":"852b0856-087c-4326-82e1-6e4aaea94ec8","Type":"ContainerStarted","Data":"825983a9d6bf351b5198b748b8b195688a3c3561f3c6b90cc5af4cf44cf7db67"}
Jul 23 19:01:42.965784 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.965762    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787859376"
Jul 23 19:01:42.965904 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:42.965814    2561 scope.go:117] "RemoveContainer" containerID="36e0c780f68897b6a21631fc6843c690eff2fb25e0c76f192d7eff9e40f259ab"
Jul 23 19:01:43.972773 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:43.972741    2561 generic.go:356] "Generic (PLEG): container finished" podID="852b0856-087c-4326-82e1-6e4aaea94ec8" containerID="db2c313b8c388554ae576ce90e670f04e58fc78f84c81a01ebab407c9f8352aa" exitCode=0
Jul 23 19:01:43.973208 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:43.972830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787858992" event={"ID":"852b0856-087c-4326-82e1-6e4aaea94ec8","Type":"ContainerDied","Data":"db2c313b8c388554ae576ce90e670f04e58fc78f84c81a01ebab407c9f8352aa"}
Jul 23 19:01:44.826024 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:44.825995    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 19:01:44.835185 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:44.835164    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm4gn_d9052287-a150-41b0-b0ef-82313217f2e7/ovn-acl-logging/0.log"
Jul 23 19:01:45.018889 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.018860    2561 scope.go:117] "RemoveContainer" containerID="c4a7caf8be2f358cf0471f13180ca02d9e6c6931e604128b4a33f8b03e34c190"
Jul 23 19:01:45.027370 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.027351    2561 scope.go:117] "RemoveContainer" containerID="8757b879b89bbb8c5f69e3a393dca6004be35dff0dbe95554d9499b24309fa49"
Jul 23 19:01:45.135235 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.135212    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787858992"
Jul 23 19:01:45.162172 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.162144    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7434-140634787858992_852b0856-087c-4326-82e1-6e4aaea94ec8/test-curl-7434-140634787858992/0.log"
Jul 23 19:01:45.189126 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.189093    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7434-140634787858992"]
Jul 23 19:01:45.192017 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.191993    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7434-140634787858992"]
Jul 23 19:01:45.272460 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.272429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shhbw\" (UniqueName: \"kubernetes.io/projected/852b0856-087c-4326-82e1-6e4aaea94ec8-kube-api-access-shhbw\") pod \"852b0856-087c-4326-82e1-6e4aaea94ec8\" (UID: \"852b0856-087c-4326-82e1-6e4aaea94ec8\") "
Jul 23 19:01:45.274427 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.274398    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852b0856-087c-4326-82e1-6e4aaea94ec8-kube-api-access-shhbw" (OuterVolumeSpecName: "kube-api-access-shhbw") pod "852b0856-087c-4326-82e1-6e4aaea94ec8" (UID: "852b0856-087c-4326-82e1-6e4aaea94ec8"). InnerVolumeSpecName "kube-api-access-shhbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:01:45.350596 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.350566    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:45.351038 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.351016    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="852b0856-087c-4326-82e1-6e4aaea94ec8" containerName="test-curl-7434-140634787858992"
Jul 23 19:01:45.351038 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.351035    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="852b0856-087c-4326-82e1-6e4aaea94ec8" containerName="test-curl-7434-140634787858992"
Jul 23 19:01:45.351205 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.351111    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="852b0856-087c-4326-82e1-6e4aaea94ec8" containerName="test-curl-7434-140634787858992"
Jul 23 19:01:45.354394 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.354373    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:45.360940 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.360915    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:45.373301 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.373185    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhbw\" (UniqueName: \"kubernetes.io/projected/852b0856-087c-4326-82e1-6e4aaea94ec8-kube-api-access-shhbw\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:45.473947 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.473909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jxl\" (UniqueName: \"kubernetes.io/projected/e36b1e09-5216-44f7-97d5-9476e6a5e20e-kube-api-access-l9jxl\") pod \"test-curl-7434-140634787632736\" (UID: \"e36b1e09-5216-44f7-97d5-9476e6a5e20e\") " pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:45.574595 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.574554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jxl\" (UniqueName: \"kubernetes.io/projected/e36b1e09-5216-44f7-97d5-9476e6a5e20e-kube-api-access-l9jxl\") pod \"test-curl-7434-140634787632736\" (UID: \"e36b1e09-5216-44f7-97d5-9476e6a5e20e\") " pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:45.584040 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.584010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jxl\" (UniqueName: \"kubernetes.io/projected/e36b1e09-5216-44f7-97d5-9476e6a5e20e-kube-api-access-l9jxl\") pod \"test-curl-7434-140634787632736\" (UID: \"e36b1e09-5216-44f7-97d5-9476e6a5e20e\") " pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:45.673013 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.672924    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:45.806351 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.806328    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:45.808150 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:01:45.808107    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode36b1e09_5216_44f7_97d5_9476e6a5e20e.slice/crio-ca8008fc560bc23cf643a9df3a4a5336b96c6f0d71bea0ce9d490d0170dd8e60 WatchSource:0}: Error finding container ca8008fc560bc23cf643a9df3a4a5336b96c6f0d71bea0ce9d490d0170dd8e60: Status 404 returned error can't find the container with id ca8008fc560bc23cf643a9df3a4a5336b96c6f0d71bea0ce9d490d0170dd8e60
Jul 23 19:01:45.985935 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.985906    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787858992"
Jul 23 19:01:45.986126 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.985937    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="825983a9d6bf351b5198b748b8b195688a3c3561f3c6b90cc5af4cf44cf7db67"
Jul 23 19:01:45.987149 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:45.987125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787632736" event={"ID":"e36b1e09-5216-44f7-97d5-9476e6a5e20e","Type":"ContainerStarted","Data":"ca8008fc560bc23cf643a9df3a4a5336b96c6f0d71bea0ce9d490d0170dd8e60"}
Jul 23 19:01:45.997376 ip-10-0-140-91 kubenswrapper[2561]: E0723 19:01:45.997353    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7434-140634787858992\" is forbidden: User \"system:node:ip-10-0-140-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-91.ec2.internal' and this object" podUID="852b0856-087c-4326-82e1-6e4aaea94ec8" pod="opendatahub/test-curl-7434-140634787858992"
Jul 23 19:01:46.761763 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:46.761729    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="852b0856-087c-4326-82e1-6e4aaea94ec8" path="/var/lib/kubelet/pods/852b0856-087c-4326-82e1-6e4aaea94ec8/volumes"
Jul 23 19:01:46.993799 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:46.993764    2561 generic.go:356] "Generic (PLEG): container finished" podID="e36b1e09-5216-44f7-97d5-9476e6a5e20e" containerID="b2e78f2ae5f045be896f0ea6dfe19430336e6adf4ca1b1736b369b272714664f" exitCode=0
Jul 23 19:01:46.993973 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:46.993813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787632736" event={"ID":"e36b1e09-5216-44f7-97d5-9476e6a5e20e","Type":"ContainerDied","Data":"b2e78f2ae5f045be896f0ea6dfe19430336e6adf4ca1b1736b369b272714664f"}
Jul 23 19:01:48.131592 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.131568    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:48.151712 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.151688    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7434-140634787632736_e36b1e09-5216-44f7-97d5-9476e6a5e20e/test-curl-7434-140634787632736/0.log"
Jul 23 19:01:48.175002 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.174970    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:48.178987 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.178963    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:48.200720 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.200699    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9jxl\" (UniqueName: \"kubernetes.io/projected/e36b1e09-5216-44f7-97d5-9476e6a5e20e-kube-api-access-l9jxl\") pod \"e36b1e09-5216-44f7-97d5-9476e6a5e20e\" (UID: \"e36b1e09-5216-44f7-97d5-9476e6a5e20e\") "
Jul 23 19:01:48.202673 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.202650    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36b1e09-5216-44f7-97d5-9476e6a5e20e-kube-api-access-l9jxl" (OuterVolumeSpecName: "kube-api-access-l9jxl") pod "e36b1e09-5216-44f7-97d5-9476e6a5e20e" (UID: "e36b1e09-5216-44f7-97d5-9476e6a5e20e"). InnerVolumeSpecName "kube-api-access-l9jxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:01:48.302366 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.302271    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9jxl\" (UniqueName: \"kubernetes.io/projected/e36b1e09-5216-44f7-97d5-9476e6a5e20e-kube-api-access-l9jxl\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:48.306694 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.306669    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:48.307052 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.307040    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36b1e09-5216-44f7-97d5-9476e6a5e20e" containerName="test-curl-7434-140634787632736"
Jul 23 19:01:48.307094 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.307054    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36b1e09-5216-44f7-97d5-9476e6a5e20e" containerName="test-curl-7434-140634787632736"
Jul 23 19:01:48.307160 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.307150    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36b1e09-5216-44f7-97d5-9476e6a5e20e" containerName="test-curl-7434-140634787632736"
Jul 23 19:01:48.310292 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.310265    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:48.317555 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.317531    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:48.403355 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.403319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrk6\" (UniqueName: \"kubernetes.io/projected/4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa-kube-api-access-xmrk6\") pod \"test-curl-7434-140634787632736\" (UID: \"4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa\") " pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:48.503842 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.503807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrk6\" (UniqueName: \"kubernetes.io/projected/4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa-kube-api-access-xmrk6\") pod \"test-curl-7434-140634787632736\" (UID: \"4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa\") " pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:48.512168 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.512132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrk6\" (UniqueName: \"kubernetes.io/projected/4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa-kube-api-access-xmrk6\") pod \"test-curl-7434-140634787632736\" (UID: \"4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa\") " pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:48.621270 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.621176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:48.761870 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.761836    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e36b1e09-5216-44f7-97d5-9476e6a5e20e" path="/var/lib/kubelet/pods/e36b1e09-5216-44f7-97d5-9476e6a5e20e/volumes"
Jul 23 19:01:48.958913 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:48.958889    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:48.960206 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:01:48.960168    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e5829b7_6619_4be1_b5f9_cd2d91d6c6aa.slice/crio-d686165efdcb93285a4b2946818c811799473abcb5ea9469e6fc9272ec245f31 WatchSource:0}: Error finding container d686165efdcb93285a4b2946818c811799473abcb5ea9469e6fc9272ec245f31: Status 404 returned error can't find the container with id d686165efdcb93285a4b2946818c811799473abcb5ea9469e6fc9272ec245f31
Jul 23 19:01:49.005190 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:49.005160    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787632736" event={"ID":"4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa","Type":"ContainerStarted","Data":"d686165efdcb93285a4b2946818c811799473abcb5ea9469e6fc9272ec245f31"}
Jul 23 19:01:49.006500 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:49.006483    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:49.006581 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:49.006528    2561 scope.go:117] "RemoveContainer" containerID="b2e78f2ae5f045be896f0ea6dfe19430336e6adf4ca1b1736b369b272714664f"
Jul 23 19:01:50.013418 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:50.013382    2561 generic.go:356] "Generic (PLEG): container finished" podID="4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa" containerID="a6629438c5b8ae4809dfeeae7fda76388bc209eea93c679e5c46dfb3a915b220" exitCode=0
Jul 23 19:01:50.013891 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:50.013464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787632736" event={"ID":"4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa","Type":"ContainerDied","Data":"a6629438c5b8ae4809dfeeae7fda76388bc209eea93c679e5c46dfb3a915b220"}
Jul 23 19:01:50.291632 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:50.291553    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:01:51.158068 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:51.158042    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:51.191843 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:51.191815    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7434-140634787632736_4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa/test-curl-7434-140634787632736/0.log"
Jul 23 19:01:51.220975 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:51.220943    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:51.226956 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:51.226932    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7434-140634787632736"]
Jul 23 19:01:51.230892 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:51.230869    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmrk6\" (UniqueName: \"kubernetes.io/projected/4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa-kube-api-access-xmrk6\") pod \"4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa\" (UID: \"4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa\") "
Jul 23 19:01:51.232920 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:51.232886    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa-kube-api-access-xmrk6" (OuterVolumeSpecName: "kube-api-access-xmrk6") pod "4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa" (UID: "4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa"). InnerVolumeSpecName "kube-api-access-xmrk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:01:51.331912 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:51.331834    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmrk6\" (UniqueName: \"kubernetes.io/projected/4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa-kube-api-access-xmrk6\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:52.026832 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:52.026807    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d686165efdcb93285a4b2946818c811799473abcb5ea9469e6fc9272ec245f31"
Jul 23 19:01:52.026832 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:52.026820    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:52.036533 ip-10-0-140-91 kubenswrapper[2561]: E0723 19:01:52.036497    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7434-140634787632736\" is forbidden: User \"system:node:ip-10-0-140-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-91.ec2.internal' and this object" podUID="4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa" pod="opendatahub/test-curl-7434-140634787632736"
Jul 23 19:01:52.762536 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:52.762503    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa" path="/var/lib/kubelet/pods/4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa/volumes"
Jul 23 19:01:52.940267 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:52.940207    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj" podUID="77c846ef-1765-4a1a-a137-cb08f4385ccf" containerName="main" containerID="cri-o://73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41" gracePeriod=45
Jul 23 19:01:53.179805 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.179780    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:01:53.249126 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249099    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc7tp\" (UniqueName: \"kubernetes.io/projected/77c846ef-1765-4a1a-a137-cb08f4385ccf-kube-api-access-rc7tp\") pod \"77c846ef-1765-4a1a-a137-cb08f4385ccf\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") "
Jul 23 19:01:53.249332 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-tmp-dir\") pod \"77c846ef-1765-4a1a-a137-cb08f4385ccf\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") "
Jul 23 19:01:53.249332 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249161    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-model-cache\") pod \"77c846ef-1765-4a1a-a137-cb08f4385ccf\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") "
Jul 23 19:01:53.249332 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249312    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-dshm\") pod \"77c846ef-1765-4a1a-a137-cb08f4385ccf\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") "
Jul 23 19:01:53.249513 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249380    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c846ef-1765-4a1a-a137-cb08f4385ccf-tls-certs\") pod \"77c846ef-1765-4a1a-a137-cb08f4385ccf\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") "
Jul 23 19:01:53.249513 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249413    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77c846ef-1765-4a1a-a137-cb08f4385ccf" (UID: "77c846ef-1765-4a1a-a137-cb08f4385ccf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:01:53.249513 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249430    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-model-cache" (OuterVolumeSpecName: "model-cache") pod "77c846ef-1765-4a1a-a137-cb08f4385ccf" (UID: "77c846ef-1765-4a1a-a137-cb08f4385ccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:01:53.249513 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249447    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-home\") pod \"77c846ef-1765-4a1a-a137-cb08f4385ccf\" (UID: \"77c846ef-1765-4a1a-a137-cb08f4385ccf\") "
Jul 23 19:01:53.249707 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249673    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-home" (OuterVolumeSpecName: "home") pod "77c846ef-1765-4a1a-a137-cb08f4385ccf" (UID: "77c846ef-1765-4a1a-a137-cb08f4385ccf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:01:53.249760 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249723    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-tmp-dir\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:53.249760 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.249736    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-model-cache\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:53.251309 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.251272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-dshm" (OuterVolumeSpecName: "dshm") pod "77c846ef-1765-4a1a-a137-cb08f4385ccf" (UID: "77c846ef-1765-4a1a-a137-cb08f4385ccf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:01:53.251416 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.251329    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c846ef-1765-4a1a-a137-cb08f4385ccf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77c846ef-1765-4a1a-a137-cb08f4385ccf" (UID: "77c846ef-1765-4a1a-a137-cb08f4385ccf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 19:01:53.251416 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.251384    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c846ef-1765-4a1a-a137-cb08f4385ccf-kube-api-access-rc7tp" (OuterVolumeSpecName: "kube-api-access-rc7tp") pod "77c846ef-1765-4a1a-a137-cb08f4385ccf" (UID: "77c846ef-1765-4a1a-a137-cb08f4385ccf"). InnerVolumeSpecName "kube-api-access-rc7tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:01:53.351102 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.351021    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc7tp\" (UniqueName: \"kubernetes.io/projected/77c846ef-1765-4a1a-a137-cb08f4385ccf-kube-api-access-rc7tp\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:53.351102 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.351051    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-dshm\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:53.351102 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.351061    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c846ef-1765-4a1a-a137-cb08f4385ccf-tls-certs\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:53.351102 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.351069    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c846ef-1765-4a1a-a137-cb08f4385ccf-home\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:53.709648 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.709605    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf" podUID="5fb16b2f-76c2-4886-9686-ad38d1821981" containerName="main" containerID="cri-o://58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03" gracePeriod=45
Jul 23 19:01:53.948823 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:53.948797    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:54.039570 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.039488    2561 generic.go:356] "Generic (PLEG): container finished" podID="5fb16b2f-76c2-4886-9686-ad38d1821981" containerID="58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03" exitCode=0
Jul 23 19:01:54.039570 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.039527    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"
Jul 23 19:01:54.039800 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.039570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf" event={"ID":"5fb16b2f-76c2-4886-9686-ad38d1821981","Type":"ContainerDied","Data":"58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03"}
Jul 23 19:01:54.039800 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.039610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf" event={"ID":"5fb16b2f-76c2-4886-9686-ad38d1821981","Type":"ContainerDied","Data":"a5a700478ccf6cc1739cf3383822bc3ce47e5287c5c6d0932047d72ed1754695"}
Jul 23 19:01:54.039800 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.039634    2561 scope.go:117] "RemoveContainer" containerID="58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03"
Jul 23 19:01:54.040919 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.040900    2561 generic.go:356] "Generic (PLEG): container finished" podID="77c846ef-1765-4a1a-a137-cb08f4385ccf" containerID="73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41" exitCode=0
Jul 23 19:01:54.041004 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.040935    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"
Jul 23 19:01:54.041004 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.040991    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj" event={"ID":"77c846ef-1765-4a1a-a137-cb08f4385ccf","Type":"ContainerDied","Data":"73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41"}
Jul 23 19:01:54.041096 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.041030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj" event={"ID":"77c846ef-1765-4a1a-a137-cb08f4385ccf","Type":"ContainerDied","Data":"3e826abb40aaf4ccc11fc188c996a574eb4c9299bde99e21720d79df60a189e4"}
Jul 23 19:01:54.048681 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.048630    2561 scope.go:117] "RemoveContainer" containerID="58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03"
Jul 23 19:01:54.048993 ip-10-0-140-91 kubenswrapper[2561]: E0723 19:01:54.048960    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03\": container with ID starting with 58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03 not found: ID does not exist" containerID="58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03"
Jul 23 19:01:54.049110 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.049001    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03"} err="failed to get container status \"58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03\": rpc error: code = NotFound desc = could not find container \"58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03\": container with ID starting with 58a6a567e8df8d06893bdef6c2e443703f975560a27f7ddd54fb5cb20835ba03 not found: ID does not exist"
Jul 23 19:01:54.049110 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.049023    2561 scope.go:117] "RemoveContainer" containerID="73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41"
Jul 23 19:01:54.057821 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.057793    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv57k\" (UniqueName: \"kubernetes.io/projected/5fb16b2f-76c2-4886-9686-ad38d1821981-kube-api-access-wv57k\") pod \"5fb16b2f-76c2-4886-9686-ad38d1821981\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") "
Jul 23 19:01:54.057897 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.057851    2561 scope.go:117] "RemoveContainer" containerID="73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41"
Jul 23 19:01:54.057897 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.057867    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-model-cache\") pod \"5fb16b2f-76c2-4886-9686-ad38d1821981\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") "
Jul 23 19:01:54.058003 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.057974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-dshm\") pod \"5fb16b2f-76c2-4886-9686-ad38d1821981\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") "
Jul 23 19:01:54.058071 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.058015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb16b2f-76c2-4886-9686-ad38d1821981-tls-certs\") pod \"5fb16b2f-76c2-4886-9686-ad38d1821981\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") "
Jul 23 19:01:54.058071 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.058051    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-home\") pod \"5fb16b2f-76c2-4886-9686-ad38d1821981\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") "
Jul 23 19:01:54.058176 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.058077    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-tmp-dir\") pod \"5fb16b2f-76c2-4886-9686-ad38d1821981\" (UID: \"5fb16b2f-76c2-4886-9686-ad38d1821981\") "
Jul 23 19:01:54.058176 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.058100    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fb16b2f-76c2-4886-9686-ad38d1821981" (UID: "5fb16b2f-76c2-4886-9686-ad38d1821981"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:01:54.058311 ip-10-0-140-91 kubenswrapper[2561]: E0723 19:01:54.058187    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41\": container with ID starting with 73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41 not found: ID does not exist" containerID="73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41"
Jul 23 19:01:54.058311 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.058217    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41"} err="failed to get container status \"73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41\": rpc error: code = NotFound desc = could not find container \"73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41\": container with ID starting with 73bc81442ae3214d4d3f29d3bc2e3480de16a1b0f87448de05a0673771d5ff41 not found: ID does not exist"
Jul 23 19:01:54.058496 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.058477    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-home" (OuterVolumeSpecName: "home") pod "5fb16b2f-76c2-4886-9686-ad38d1821981" (UID: "5fb16b2f-76c2-4886-9686-ad38d1821981"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:01:54.058496 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.058491    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-model-cache\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:54.058712 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.058690    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fb16b2f-76c2-4886-9686-ad38d1821981" (UID: "5fb16b2f-76c2-4886-9686-ad38d1821981"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:01:54.060008 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.059984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb16b2f-76c2-4886-9686-ad38d1821981-kube-api-access-wv57k" (OuterVolumeSpecName: "kube-api-access-wv57k") pod "5fb16b2f-76c2-4886-9686-ad38d1821981" (UID: "5fb16b2f-76c2-4886-9686-ad38d1821981"). InnerVolumeSpecName "kube-api-access-wv57k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:01:54.060129 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.060010    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb16b2f-76c2-4886-9686-ad38d1821981-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fb16b2f-76c2-4886-9686-ad38d1821981" (UID: "5fb16b2f-76c2-4886-9686-ad38d1821981"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 19:01:54.060129 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.060092    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-dshm" (OuterVolumeSpecName: "dshm") pod "5fb16b2f-76c2-4886-9686-ad38d1821981" (UID: "5fb16b2f-76c2-4886-9686-ad38d1821981"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 19:01:54.064570 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.064546    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"]
Jul 23 19:01:54.067677 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.067652    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/rate-test-model-3f12e9-kserve-b77598b99-qsdxj"]
Jul 23 19:01:54.159055 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.159025    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv57k\" (UniqueName: \"kubernetes.io/projected/5fb16b2f-76c2-4886-9686-ad38d1821981-kube-api-access-wv57k\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:54.159055 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.159051    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-dshm\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:54.159055 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.159061    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb16b2f-76c2-4886-9686-ad38d1821981-tls-certs\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:54.159309 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.159069    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-home\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:54.159309 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.159077    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb16b2f-76c2-4886-9686-ad38d1821981-tmp-dir\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:54.362208 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.362181    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"]
Jul 23 19:01:54.364890 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.364864    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/rate-test-model-60de5a-kserve-55f664484c-m4dpf"]
Jul 23 19:01:54.382614 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.382584    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7434-140634787633744"]
Jul 23 19:01:54.383013 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.382999    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb16b2f-76c2-4886-9686-ad38d1821981" containerName="main"
Jul 23 19:01:54.383069 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.383014    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb16b2f-76c2-4886-9686-ad38d1821981" containerName="main"
Jul 23 19:01:54.383069 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.383025    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c846ef-1765-4a1a-a137-cb08f4385ccf" containerName="main"
Jul 23 19:01:54.383069 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.383031    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c846ef-1765-4a1a-a137-cb08f4385ccf" containerName="main"
Jul 23 19:01:54.383069 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.383038    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa" containerName="test-curl-7434-140634787632736"
Jul 23 19:01:54.383069 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.383044    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa" containerName="test-curl-7434-140634787632736"
Jul 23 19:01:54.385696 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.383706    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c846ef-1765-4a1a-a137-cb08f4385ccf" containerName="main"
Jul 23 19:01:54.385696 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.383745    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5829b7-6619-4be1-b5f9-cd2d91d6c6aa" containerName="test-curl-7434-140634787632736"
Jul 23 19:01:54.385696 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.383759    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb16b2f-76c2-4886-9686-ad38d1821981" containerName="main"
Jul 23 19:01:54.389320 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.389217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787633744"
Jul 23 19:01:54.392385 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.392348    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-92csk\""
Jul 23 19:01:54.395122 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.395100    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787633744"]
Jul 23 19:01:54.562963 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.562935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9bf\" (UniqueName: \"kubernetes.io/projected/33d5eef8-d152-4f8b-8441-69219d90a036-kube-api-access-4f9bf\") pod \"test-curl-7434-140634787633744\" (UID: \"33d5eef8-d152-4f8b-8441-69219d90a036\") " pod="opendatahub/test-curl-7434-140634787633744"
Jul 23 19:01:54.663479 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.663396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9bf\" (UniqueName: \"kubernetes.io/projected/33d5eef8-d152-4f8b-8441-69219d90a036-kube-api-access-4f9bf\") pod \"test-curl-7434-140634787633744\" (UID: \"33d5eef8-d152-4f8b-8441-69219d90a036\") " pod="opendatahub/test-curl-7434-140634787633744"
Jul 23 19:01:54.672416 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.672388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9bf\" (UniqueName: \"kubernetes.io/projected/33d5eef8-d152-4f8b-8441-69219d90a036-kube-api-access-4f9bf\") pod \"test-curl-7434-140634787633744\" (UID: \"33d5eef8-d152-4f8b-8441-69219d90a036\") " pod="opendatahub/test-curl-7434-140634787633744"
Jul 23 19:01:54.699930 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.699906    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787633744"
Jul 23 19:01:54.763570 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.763541    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fb16b2f-76c2-4886-9686-ad38d1821981" path="/var/lib/kubelet/pods/5fb16b2f-76c2-4886-9686-ad38d1821981/volumes"
Jul 23 19:01:54.763931 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.763918    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77c846ef-1765-4a1a-a137-cb08f4385ccf" path="/var/lib/kubelet/pods/77c846ef-1765-4a1a-a137-cb08f4385ccf/volumes"
Jul 23 19:01:54.827922 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:54.827897    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787633744"]
Jul 23 19:01:54.829866 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:01:54.829834    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33d5eef8_d152_4f8b_8441_69219d90a036.slice/crio-14b9b77b432b1a915edd39a72c3386cfd34b028c02a6515ddea7689690cf04d0 WatchSource:0}: Error finding container 14b9b77b432b1a915edd39a72c3386cfd34b028c02a6515ddea7689690cf04d0: Status 404 returned error can't find the container with id 14b9b77b432b1a915edd39a72c3386cfd34b028c02a6515ddea7689690cf04d0
Jul 23 19:01:55.047919 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:55.047885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787633744" event={"ID":"33d5eef8-d152-4f8b-8441-69219d90a036","Type":"ContainerStarted","Data":"14b9b77b432b1a915edd39a72c3386cfd34b028c02a6515ddea7689690cf04d0"}
Jul 23 19:01:56.055421 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:56.055386    2561 generic.go:356] "Generic (PLEG): container finished" podID="33d5eef8-d152-4f8b-8441-69219d90a036" containerID="c0d8a7379deb57fe4daed6876db7c215a914232488f6a94755857846d7b3f8f8" exitCode=0
Jul 23 19:01:56.055421 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:56.055421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787633744" event={"ID":"33d5eef8-d152-4f8b-8441-69219d90a036","Type":"ContainerDied","Data":"c0d8a7379deb57fe4daed6876db7c215a914232488f6a94755857846d7b3f8f8"}
Jul 23 19:01:57.197495 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.197468    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787633744"
Jul 23 19:01:57.219427 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.219381    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7434-140634787633744_33d5eef8-d152-4f8b-8441-69219d90a036/test-curl-7434-140634787633744/0.log"
Jul 23 19:01:57.245177 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.245142    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7434-140634787633744"]
Jul 23 19:01:57.251719 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.251686    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7434-140634787633744"]
Jul 23 19:01:57.287254 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.287225    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f9bf\" (UniqueName: \"kubernetes.io/projected/33d5eef8-d152-4f8b-8441-69219d90a036-kube-api-access-4f9bf\") pod \"33d5eef8-d152-4f8b-8441-69219d90a036\" (UID: \"33d5eef8-d152-4f8b-8441-69219d90a036\") "
Jul 23 19:01:57.289271 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.289245    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d5eef8-d152-4f8b-8441-69219d90a036-kube-api-access-4f9bf" (OuterVolumeSpecName: "kube-api-access-4f9bf") pod "33d5eef8-d152-4f8b-8441-69219d90a036" (UID: "33d5eef8-d152-4f8b-8441-69219d90a036"). InnerVolumeSpecName "kube-api-access-4f9bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:01:57.388702 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.388617    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f9bf\" (UniqueName: \"kubernetes.io/projected/33d5eef8-d152-4f8b-8441-69219d90a036-kube-api-access-4f9bf\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:01:57.444426 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.444389    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"]
Jul 23 19:01:57.444862 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.444842    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d5eef8-d152-4f8b-8441-69219d90a036" containerName="test-curl-7434-140634787633744"
Jul 23 19:01:57.444945 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.444865    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d5eef8-d152-4f8b-8441-69219d90a036" containerName="test-curl-7434-140634787633744"
Jul 23 19:01:57.445000 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.444964    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d5eef8-d152-4f8b-8441-69219d90a036" containerName="test-curl-7434-140634787633744"
Jul 23 19:01:57.447958 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.447937    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"
Jul 23 19:01:57.450586 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.450566    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 19:01:57.458946 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.458918    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"]
Jul 23 19:01:57.590779 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.590743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995f75a5-7036-408c-a0e7-4c36575f5113-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh\" (UID: \"995f75a5-7036-408c-a0e7-4c36575f5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"
Jul 23 19:01:57.692248 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.692155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995f75a5-7036-408c-a0e7-4c36575f5113-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh\" (UID: \"995f75a5-7036-408c-a0e7-4c36575f5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"
Jul 23 19:01:57.692789 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.692770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995f75a5-7036-408c-a0e7-4c36575f5113-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh\" (UID: \"995f75a5-7036-408c-a0e7-4c36575f5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"
Jul 23 19:01:57.758852 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.758815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"
Jul 23 19:01:57.892858 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:57.892824    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"]
Jul 23 19:01:57.896426 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:01:57.896403    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995f75a5_7036_408c_a0e7_4c36575f5113.slice/crio-f549c618a8953456b98bcd7d287532c79196b64af39f158429d7341bcbeb60d1 WatchSource:0}: Error finding container f549c618a8953456b98bcd7d287532c79196b64af39f158429d7341bcbeb60d1: Status 404 returned error can't find the container with id f549c618a8953456b98bcd7d287532c79196b64af39f158429d7341bcbeb60d1
Jul 23 19:01:58.070818 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:58.070787    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14b9b77b432b1a915edd39a72c3386cfd34b028c02a6515ddea7689690cf04d0"
Jul 23 19:01:58.070818 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:58.070788    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787633744"
Jul 23 19:01:58.071926 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:58.071893    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh" event={"ID":"995f75a5-7036-408c-a0e7-4c36575f5113","Type":"ContainerStarted","Data":"f549c618a8953456b98bcd7d287532c79196b64af39f158429d7341bcbeb60d1"}
Jul 23 19:01:58.081745 ip-10-0-140-91 kubenswrapper[2561]: E0723 19:01:58.081717    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7434-140634787633744\" is forbidden: User \"system:node:ip-10-0-140-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-91.ec2.internal' and this object" podUID="33d5eef8-d152-4f8b-8441-69219d90a036" pod="opendatahub/test-curl-7434-140634787633744"
Jul 23 19:01:58.761130 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:58.761092    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d5eef8-d152-4f8b-8441-69219d90a036" path="/var/lib/kubelet/pods/33d5eef8-d152-4f8b-8441-69219d90a036/volumes"
Jul 23 19:01:59.078582 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:59.078549    2561 generic.go:356] "Generic (PLEG): container finished" podID="995f75a5-7036-408c-a0e7-4c36575f5113" containerID="7da0a739968f336a2305c8f6f1196352d16633389c60bd0ca647d3cce7539b34" exitCode=0
Jul 23 19:01:59.078741 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:01:59.078592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh" event={"ID":"995f75a5-7036-408c-a0e7-4c36575f5113","Type":"ContainerDied","Data":"7da0a739968f336a2305c8f6f1196352d16633389c60bd0ca647d3cce7539b34"}
Jul 23 19:02:00.226091 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:00.226061    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"
Jul 23 19:02:00.316653 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:00.316618    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995f75a5-7036-408c-a0e7-4c36575f5113-maas-api-service-ca\") pod \"995f75a5-7036-408c-a0e7-4c36575f5113\" (UID: \"995f75a5-7036-408c-a0e7-4c36575f5113\") "
Jul 23 19:02:00.316988 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:00.316966    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995f75a5-7036-408c-a0e7-4c36575f5113-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "995f75a5-7036-408c-a0e7-4c36575f5113" (UID: "995f75a5-7036-408c-a0e7-4c36575f5113"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 19:02:00.417587 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:00.417501    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/995f75a5-7036-408c-a0e7-4c36575f5113-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:02:01.089758 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:01.089671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh" event={"ID":"995f75a5-7036-408c-a0e7-4c36575f5113","Type":"ContainerDied","Data":"f549c618a8953456b98bcd7d287532c79196b64af39f158429d7341bcbeb60d1"}
Jul 23 19:02:01.089758 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:01.089720    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f549c618a8953456b98bcd7d287532c79196b64af39f158429d7341bcbeb60d1"
Jul 23 19:02:01.089758 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:01.089679    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"
Jul 23 19:02:07.959883 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:07.959847    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"]
Jul 23 19:02:07.960370 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:07.960069    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc" podUID="36c28355-c254-4a07-ba14-d3b578c1b238" containerName="maas-api" containerID="cri-o://69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52" gracePeriod=30
Jul 23 19:02:08.010162 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.010125    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr"]
Jul 23 19:02:08.012831 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.012804    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-3f12e9-29747220-h59tr"]
Jul 23 19:02:08.408401 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.408375    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 19:02:08.485878 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.485793    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c28355-c254-4a07-ba14-d3b578c1b238-openshift-service-ca\") pod \"36c28355-c254-4a07-ba14-d3b578c1b238\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") "
Jul 23 19:02:08.485878 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.485841    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mfv7\" (UniqueName: \"kubernetes.io/projected/36c28355-c254-4a07-ba14-d3b578c1b238-kube-api-access-6mfv7\") pod \"36c28355-c254-4a07-ba14-d3b578c1b238\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") "
Jul 23 19:02:08.485878 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.485888    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c28355-c254-4a07-ba14-d3b578c1b238-maas-api-tls\") pod \"36c28355-c254-4a07-ba14-d3b578c1b238\" (UID: \"36c28355-c254-4a07-ba14-d3b578c1b238\") "
Jul 23 19:02:08.486155 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.486131    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c28355-c254-4a07-ba14-d3b578c1b238-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36c28355-c254-4a07-ba14-d3b578c1b238" (UID: "36c28355-c254-4a07-ba14-d3b578c1b238"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 19:02:08.487911 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.487886    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c28355-c254-4a07-ba14-d3b578c1b238-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36c28355-c254-4a07-ba14-d3b578c1b238" (UID: "36c28355-c254-4a07-ba14-d3b578c1b238"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 19:02:08.488080 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.488062    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c28355-c254-4a07-ba14-d3b578c1b238-kube-api-access-6mfv7" (OuterVolumeSpecName: "kube-api-access-6mfv7") pod "36c28355-c254-4a07-ba14-d3b578c1b238" (UID: "36c28355-c254-4a07-ba14-d3b578c1b238"). InnerVolumeSpecName "kube-api-access-6mfv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:02:08.586630 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.586599    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c28355-c254-4a07-ba14-d3b578c1b238-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:02:08.586630 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.586624    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mfv7\" (UniqueName: \"kubernetes.io/projected/36c28355-c254-4a07-ba14-d3b578c1b238-kube-api-access-6mfv7\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:02:08.586630 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.586633    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c28355-c254-4a07-ba14-d3b578c1b238-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:02:08.761381 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:08.761297    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea185c61-69ce-4be2-8f93-20a4c5909d86" path="/var/lib/kubelet/pods/ea185c61-69ce-4be2-8f93-20a4c5909d86/volumes"
Jul 23 19:02:09.132560 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:09.132469    2561 generic.go:356] "Generic (PLEG): container finished" podID="36c28355-c254-4a07-ba14-d3b578c1b238" containerID="69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52" exitCode=0
Jul 23 19:02:09.132560 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:09.132516    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"
Jul 23 19:02:09.132560 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:09.132551    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc" event={"ID":"36c28355-c254-4a07-ba14-d3b578c1b238","Type":"ContainerDied","Data":"69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52"}
Jul 23 19:02:09.133018 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:09.132604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc" event={"ID":"36c28355-c254-4a07-ba14-d3b578c1b238","Type":"ContainerDied","Data":"69c0086d52048474d49eb053ae863f45e4a0fe0c829dc5e741e1cc961a6fe808"}
Jul 23 19:02:09.133018 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:09.132627    2561 scope.go:117] "RemoveContainer" containerID="69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52"
Jul 23 19:02:09.141321 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:09.141309    2561 scope.go:117] "RemoveContainer" containerID="69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52"
Jul 23 19:02:09.141587 ip-10-0-140-91 kubenswrapper[2561]: E0723 19:02:09.141570    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52\": container with ID starting with 69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52 not found: ID does not exist" containerID="69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52"
Jul 23 19:02:09.141634 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:09.141595    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52"} err="failed to get container status \"69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52\": rpc error: code = NotFound desc = could not find container \"69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52\": container with ID starting with 69a73a66cd47fb402f32b176eb93db369239b47846365b365c3612eb109d8a52 not found: ID does not exist"
Jul 23 19:02:09.151980 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:09.151955    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"]
Jul 23 19:02:09.159717 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:09.159696    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f12e9-9d87c7886-2qrhc"]
Jul 23 19:02:10.761419 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:10.761385    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36c28355-c254-4a07-ba14-d3b578c1b238" path="/var/lib/kubelet/pods/36c28355-c254-4a07-ba14-d3b578c1b238/volumes"
Jul 23 19:02:11.395003 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:11.394968    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:02:12.576791 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:12.576753    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"]
Jul 23 19:02:12.582318 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:12.582277    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f12e9-5r5vh"]
Jul 23 19:02:12.760614 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:12.760581    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="995f75a5-7036-408c-a0e7-4c36575f5113" path="/var/lib/kubelet/pods/995f75a5-7036-408c-a0e7-4c36575f5113/volumes"
Jul 23 19:02:19.982902 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.982865    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"]
Jul 23 19:02:19.983383 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.983242    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="995f75a5-7036-408c-a0e7-4c36575f5113" containerName="revoke-keys"
Jul 23 19:02:19.983383 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.983252    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="995f75a5-7036-408c-a0e7-4c36575f5113" containerName="revoke-keys"
Jul 23 19:02:19.983383 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.983275    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c28355-c254-4a07-ba14-d3b578c1b238" containerName="maas-api"
Jul 23 19:02:19.983383 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.983280    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c28355-c254-4a07-ba14-d3b578c1b238" containerName="maas-api"
Jul 23 19:02:19.983383 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.983349    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="995f75a5-7036-408c-a0e7-4c36575f5113" containerName="revoke-keys"
Jul 23 19:02:19.983383 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.983360    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="36c28355-c254-4a07-ba14-d3b578c1b238" containerName="maas-api"
Jul 23 19:02:19.988308 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.988274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"
Jul 23 19:02:19.993512 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.993488    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nr4sd\""
Jul 23 19:02:19.997443 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:19.997421    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"]
Jul 23 19:02:20.086174 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:20.086141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abd3104-52c5-4c8f-bf19-d0b47752d4e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58\" (UID: \"5abd3104-52c5-4c8f-bf19-d0b47752d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"
Jul 23 19:02:20.187677 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:20.187635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abd3104-52c5-4c8f-bf19-d0b47752d4e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58\" (UID: \"5abd3104-52c5-4c8f-bf19-d0b47752d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"
Jul 23 19:02:20.188238 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:20.188220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abd3104-52c5-4c8f-bf19-d0b47752d4e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58\" (UID: \"5abd3104-52c5-4c8f-bf19-d0b47752d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"
Jul 23 19:02:20.299933 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:20.299849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"
Jul 23 19:02:20.436267 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:20.436242    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"]
Jul 23 19:02:20.438750 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:02:20.438720    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abd3104_52c5_4c8f_bf19_d0b47752d4e4.slice/crio-bdcb2deedf5c9dc584771ee4c661f46594f85e4aedcce5b52286b725c12d1c39 WatchSource:0}: Error finding container bdcb2deedf5c9dc584771ee4c661f46594f85e4aedcce5b52286b725c12d1c39: Status 404 returned error can't find the container with id bdcb2deedf5c9dc584771ee4c661f46594f85e4aedcce5b52286b725c12d1c39
Jul 23 19:02:21.200680 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:21.200642    2561 generic.go:356] "Generic (PLEG): container finished" podID="5abd3104-52c5-4c8f-bf19-d0b47752d4e4" containerID="2072a6e5aba5575f01dbeb0307250124eee76c8a31c9d1fc10e3795f45214202" exitCode=0
Jul 23 19:02:21.201018 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:21.200721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58" event={"ID":"5abd3104-52c5-4c8f-bf19-d0b47752d4e4","Type":"ContainerDied","Data":"2072a6e5aba5575f01dbeb0307250124eee76c8a31c9d1fc10e3795f45214202"}
Jul 23 19:02:21.201018 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:21.200771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58" event={"ID":"5abd3104-52c5-4c8f-bf19-d0b47752d4e4","Type":"ContainerStarted","Data":"bdcb2deedf5c9dc584771ee4c661f46594f85e4aedcce5b52286b725c12d1c39"}
Jul 23 19:02:22.342779 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:22.342752    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"
Jul 23 19:02:22.409793 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:22.409763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abd3104-52c5-4c8f-bf19-d0b47752d4e4-maas-api-service-ca\") pod \"5abd3104-52c5-4c8f-bf19-d0b47752d4e4\" (UID: \"5abd3104-52c5-4c8f-bf19-d0b47752d4e4\") "
Jul 23 19:02:22.410145 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:22.410126    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abd3104-52c5-4c8f-bf19-d0b47752d4e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5abd3104-52c5-4c8f-bf19-d0b47752d4e4" (UID: "5abd3104-52c5-4c8f-bf19-d0b47752d4e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 19:02:22.510513 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:22.510437    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abd3104-52c5-4c8f-bf19-d0b47752d4e4-maas-api-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:02:23.212485 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:23.212460    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"
Jul 23 19:02:23.212642 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:23.212452    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58" event={"ID":"5abd3104-52c5-4c8f-bf19-d0b47752d4e4","Type":"ContainerDied","Data":"bdcb2deedf5c9dc584771ee4c661f46594f85e4aedcce5b52286b725c12d1c39"}
Jul 23 19:02:23.212642 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:23.212613    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdcb2deedf5c9dc584771ee4c661f46594f85e4aedcce5b52286b725c12d1c39"
Jul 23 19:02:30.576342 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.576308    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"]
Jul 23 19:02:30.576757 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.576527    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx" podUID="312ce815-1a40-4a2a-97fb-1dde041829db" containerName="maas-api" containerID="cri-o://c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08" gracePeriod=30
Jul 23 19:02:30.614568 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.614540    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2"]
Jul 23 19:02:30.621351 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.621328    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-60de5a-29747220-nm8w2"]
Jul 23 19:02:30.761349 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.761318    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9a8529c-4c90-4d65-9ead-4f5e9e253f93" path="/var/lib/kubelet/pods/a9a8529c-4c90-4d65-9ead-4f5e9e253f93/volumes"
Jul 23 19:02:30.820190 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.820165    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 19:02:30.886586 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.886504    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312ce815-1a40-4a2a-97fb-1dde041829db-maas-api-tls\") pod \"312ce815-1a40-4a2a-97fb-1dde041829db\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") "
Jul 23 19:02:30.886586 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.886575    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ce815-1a40-4a2a-97fb-1dde041829db-openshift-service-ca\") pod \"312ce815-1a40-4a2a-97fb-1dde041829db\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") "
Jul 23 19:02:30.886808 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.886604    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrtf6\" (UniqueName: \"kubernetes.io/projected/312ce815-1a40-4a2a-97fb-1dde041829db-kube-api-access-lrtf6\") pod \"312ce815-1a40-4a2a-97fb-1dde041829db\" (UID: \"312ce815-1a40-4a2a-97fb-1dde041829db\") "
Jul 23 19:02:30.886978 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.886927    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312ce815-1a40-4a2a-97fb-1dde041829db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "312ce815-1a40-4a2a-97fb-1dde041829db" (UID: "312ce815-1a40-4a2a-97fb-1dde041829db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 19:02:30.887135 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.887108    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312ce815-1a40-4a2a-97fb-1dde041829db-openshift-service-ca\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:02:30.888708 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.888686    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/312ce815-1a40-4a2a-97fb-1dde041829db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "312ce815-1a40-4a2a-97fb-1dde041829db" (UID: "312ce815-1a40-4a2a-97fb-1dde041829db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 19:02:30.888789 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.888730    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312ce815-1a40-4a2a-97fb-1dde041829db-kube-api-access-lrtf6" (OuterVolumeSpecName: "kube-api-access-lrtf6") pod "312ce815-1a40-4a2a-97fb-1dde041829db" (UID: "312ce815-1a40-4a2a-97fb-1dde041829db"). InnerVolumeSpecName "kube-api-access-lrtf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 19:02:30.988184 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.988151    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312ce815-1a40-4a2a-97fb-1dde041829db-maas-api-tls\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:02:30.988184 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:30.988179    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrtf6\" (UniqueName: \"kubernetes.io/projected/312ce815-1a40-4a2a-97fb-1dde041829db-kube-api-access-lrtf6\") on node \"ip-10-0-140-91.ec2.internal\" DevicePath \"\""
Jul 23 19:02:31.254761 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:31.254727    2561 generic.go:356] "Generic (PLEG): container finished" podID="312ce815-1a40-4a2a-97fb-1dde041829db" containerID="c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08" exitCode=0
Jul 23 19:02:31.254939 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:31.254772    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"
Jul 23 19:02:31.254939 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:31.254817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx" event={"ID":"312ce815-1a40-4a2a-97fb-1dde041829db","Type":"ContainerDied","Data":"c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08"}
Jul 23 19:02:31.254939 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:31.254875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx" event={"ID":"312ce815-1a40-4a2a-97fb-1dde041829db","Type":"ContainerDied","Data":"e8e89d7dafe1f84a332a009bf30fdf40492e6f31e8941943dec929bdabfae234"}
Jul 23 19:02:31.254939 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:31.254905    2561 scope.go:117] "RemoveContainer" containerID="c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08"
Jul 23 19:02:31.266926 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:31.266898    2561 scope.go:117] "RemoveContainer" containerID="c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08"
Jul 23 19:02:31.267385 ip-10-0-140-91 kubenswrapper[2561]: E0723 19:02:31.267363    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08\": container with ID starting with c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08 not found: ID does not exist" containerID="c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08"
Jul 23 19:02:31.267464 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:31.267396    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08"} err="failed to get container status \"c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08\": rpc error: code = NotFound desc = could not find container \"c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08\": container with ID starting with c2cb763529b1387c66256dc48c53d7b4b007a6a7a7c9eaf022cec8e99c760a08 not found: ID does not exist"
Jul 23 19:02:31.284550 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:31.284527    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"]
Jul 23 19:02:31.291940 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:31.291918    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-60de5a-57d8c988b-x7tdx"]
Jul 23 19:02:32.760921 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:32.760889    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="312ce815-1a40-4a2a-97fb-1dde041829db" path="/var/lib/kubelet/pods/312ce815-1a40-4a2a-97fb-1dde041829db/volumes"
Jul 23 19:02:33.794158 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:33.794122    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qplmq"]
Jul 23 19:02:35.075156 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:35.075118    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"]
Jul 23 19:02:35.077848 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:35.077824    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-60de5a-bmt58"]
Jul 23 19:02:36.760728 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:36.760692    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5abd3104-52c5-4c8f-bf19-d0b47752d4e4" path="/var/lib/kubelet/pods/5abd3104-52c5-4c8f-bf19-d0b47752d4e4/volumes"
Jul 23 19:02:48.782044 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:48.782012    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bdbd87db-hwlbr_37dedd14-286c-4b6e-a696-398b5b875b8e/copy-manifests/0.log"
Jul 23 19:02:48.794412 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:48.794388    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bdbd87db-hwlbr_37dedd14-286c-4b6e-a696-398b5b875b8e/manager/0.log"
Jul 23 19:02:49.518761 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:49.518735    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747205-xt8k5_9f7d584f-e615-47d4-bfe5-d532c55b7068/cleanup/0.log"
Jul 23 19:02:49.628473 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:49.628445    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747220-tjtgs_f1b03216-3eff-4435-975c-c918ae179d84/cleanup/0.log"
Jul 23 19:02:51.255049 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:51.255018    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vj62c_3e31c93f-895b-4ac0-884d-bd034db9b08f/manager/0.log"
Jul 23 19:02:51.709647 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:51.709619    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-qplmq_b473aed2-ddf8-490b-9ef3-4993d9df05f6/limitador/0.log"
Jul 23 19:02:51.825866 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:51.825839    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zg825_39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1/manager/0.log"
Jul 23 19:02:52.413388 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:52.413360    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b75b7b5f6-598qm_2f75f153-7e86-4dc0-ad37-024ac81f3fba/kube-auth-proxy/0.log"
Jul 23 19:02:59.552815 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.552784    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-48l67/must-gather-gp2zd"]
Jul 23 19:02:59.553198 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.553184    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5abd3104-52c5-4c8f-bf19-d0b47752d4e4" containerName="revoke-keys"
Jul 23 19:02:59.553198 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.553199    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5abd3104-52c5-4c8f-bf19-d0b47752d4e4" containerName="revoke-keys"
Jul 23 19:02:59.553339 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.553217    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312ce815-1a40-4a2a-97fb-1dde041829db" containerName="maas-api"
Jul 23 19:02:59.553339 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.553223    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="312ce815-1a40-4a2a-97fb-1dde041829db" containerName="maas-api"
Jul 23 19:02:59.553339 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.553310    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="312ce815-1a40-4a2a-97fb-1dde041829db" containerName="maas-api"
Jul 23 19:02:59.553339 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.553324    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5abd3104-52c5-4c8f-bf19-d0b47752d4e4" containerName="revoke-keys"
Jul 23 19:02:59.556464 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.556443    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48l67/must-gather-gp2zd"
Jul 23 19:02:59.558754 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.558733    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48l67\"/\"kube-root-ca.crt\""
Jul 23 19:02:59.558977 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.558955    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48l67\"/\"openshift-service-ca.crt\""
Jul 23 19:02:59.559626 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.559608    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-48l67\"/\"default-dockercfg-r5r2t\""
Jul 23 19:02:59.566590 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.566564    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48l67/must-gather-gp2zd"]
Jul 23 19:02:59.634791 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.634756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2cb995fb-3e65-4d75-b874-c33ef53e4f02-must-gather-output\") pod \"must-gather-gp2zd\" (UID: \"2cb995fb-3e65-4d75-b874-c33ef53e4f02\") " pod="openshift-must-gather-48l67/must-gather-gp2zd"
Jul 23 19:02:59.634955 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.634828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526zs\" (UniqueName: \"kubernetes.io/projected/2cb995fb-3e65-4d75-b874-c33ef53e4f02-kube-api-access-526zs\") pod \"must-gather-gp2zd\" (UID: \"2cb995fb-3e65-4d75-b874-c33ef53e4f02\") " pod="openshift-must-gather-48l67/must-gather-gp2zd"
Jul 23 19:02:59.735523 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.735489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2cb995fb-3e65-4d75-b874-c33ef53e4f02-must-gather-output\") pod \"must-gather-gp2zd\" (UID: \"2cb995fb-3e65-4d75-b874-c33ef53e4f02\") " pod="openshift-must-gather-48l67/must-gather-gp2zd"
Jul 23 19:02:59.735716 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.735565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526zs\" (UniqueName: \"kubernetes.io/projected/2cb995fb-3e65-4d75-b874-c33ef53e4f02-kube-api-access-526zs\") pod \"must-gather-gp2zd\" (UID: \"2cb995fb-3e65-4d75-b874-c33ef53e4f02\") " pod="openshift-must-gather-48l67/must-gather-gp2zd"
Jul 23 19:02:59.735884 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.735861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2cb995fb-3e65-4d75-b874-c33ef53e4f02-must-gather-output\") pod \"must-gather-gp2zd\" (UID: \"2cb995fb-3e65-4d75-b874-c33ef53e4f02\") " pod="openshift-must-gather-48l67/must-gather-gp2zd"
Jul 23 19:02:59.745646 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.745627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-526zs\" (UniqueName: \"kubernetes.io/projected/2cb995fb-3e65-4d75-b874-c33ef53e4f02-kube-api-access-526zs\") pod \"must-gather-gp2zd\" (UID: \"2cb995fb-3e65-4d75-b874-c33ef53e4f02\") " pod="openshift-must-gather-48l67/must-gather-gp2zd"
Jul 23 19:02:59.866754 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.866678    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48l67/must-gather-gp2zd"
Jul 23 19:02:59.999223 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:02:59.999200    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48l67/must-gather-gp2zd"]
Jul 23 19:03:00.001649 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:03:00.001623    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb995fb_3e65_4d75_b874_c33ef53e4f02.slice/crio-f32d4a562710e0b5e2f9c6ac39273a5c36624516186c792f1e541e6a41d71ce0 WatchSource:0}: Error finding container f32d4a562710e0b5e2f9c6ac39273a5c36624516186c792f1e541e6a41d71ce0: Status 404 returned error can't find the container with id f32d4a562710e0b5e2f9c6ac39273a5c36624516186c792f1e541e6a41d71ce0
Jul 23 19:03:00.410893 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:00.410857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48l67/must-gather-gp2zd" event={"ID":"2cb995fb-3e65-4d75-b874-c33ef53e4f02","Type":"ContainerStarted","Data":"f32d4a562710e0b5e2f9c6ac39273a5c36624516186c792f1e541e6a41d71ce0"}
Jul 23 19:03:01.418250 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:01.418215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48l67/must-gather-gp2zd" event={"ID":"2cb995fb-3e65-4d75-b874-c33ef53e4f02","Type":"ContainerStarted","Data":"c14435a017d3a8ba6e15197890af3d210e979e5689d9928d3144554001b403b5"}
Jul 23 19:03:01.418694 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:01.418263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48l67/must-gather-gp2zd" event={"ID":"2cb995fb-3e65-4d75-b874-c33ef53e4f02","Type":"ContainerStarted","Data":"f2545d9e88187ff1bdb7f5c863aa9229cf4d041b31fb60514879f1d5f3bf6d02"}
Jul 23 19:03:01.435800 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:01.435752    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-48l67/must-gather-gp2zd" podStartSLOduration=1.506087572 podStartE2EDuration="2.435735239s" totalImagesPullingTime="929.647667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:03:01.43492472 +0000 UTC m=+2777.282615247" watchObservedRunningTime="2026-07-23 19:03:01.435735239 +0000 UTC m=+2777.283425765"
Jul 23 19:03:02.678978 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:02.678945    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vqch6_4ee12eab-d883-42ae-94cc-6618823fad3c/global-pull-secret-syncer/0.log"
Jul 23 19:03:02.846955 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:02.846928    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j6gpl_39bf1855-8430-4c11-a577-32be87bf9fa5/konnectivity-agent/0.log"
Jul 23 19:03:02.925226 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:02.925193    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-91.ec2.internal_292e00e11710d5a1a45e5b6a5591eaca/haproxy/0.log"
Jul 23 19:03:07.940825 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:07.940724    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vj62c_3e31c93f-895b-4ac0-884d-bd034db9b08f/manager/0.log"
Jul 23 19:03:08.210234 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:08.210209    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-qplmq_b473aed2-ddf8-490b-9ef3-4993d9df05f6/limitador/0.log"
Jul 23 19:03:08.339106 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:08.339074    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zg825_39ad9f6f-e586-4e98-bf1b-63bfc45cd8d1/manager/0.log"
Jul 23 19:03:09.887179 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:09.887077    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32de064e-90de-4385-82af-f74dbe5e8599/alertmanager/0.log"
Jul 23 19:03:09.910490 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:09.910452    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32de064e-90de-4385-82af-f74dbe5e8599/config-reloader/0.log"
Jul 23 19:03:09.937002 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:09.936966    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32de064e-90de-4385-82af-f74dbe5e8599/kube-rbac-proxy-web/0.log"
Jul 23 19:03:09.956999 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:09.956952    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32de064e-90de-4385-82af-f74dbe5e8599/kube-rbac-proxy/0.log"
Jul 23 19:03:09.980401 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:09.978768    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32de064e-90de-4385-82af-f74dbe5e8599/kube-rbac-proxy-metric/0.log"
Jul 23 19:03:10.003379 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.003346    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32de064e-90de-4385-82af-f74dbe5e8599/prom-label-proxy/0.log"
Jul 23 19:03:10.026746 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.026711    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32de064e-90de-4385-82af-f74dbe5e8599/init-config-reloader/0.log"
Jul 23 19:03:10.102485 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.102409    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vlgm7_943a1fe3-2eb3-4b63-8de0-edadaf8c5aed/kube-state-metrics/0.log"
Jul 23 19:03:10.121245 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.121217    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vlgm7_943a1fe3-2eb3-4b63-8de0-edadaf8c5aed/kube-rbac-proxy-main/0.log"
Jul 23 19:03:10.147130 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.147053    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vlgm7_943a1fe3-2eb3-4b63-8de0-edadaf8c5aed/kube-rbac-proxy-self/0.log"
Jul 23 19:03:10.175679 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.175648    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c58cdbd8-f88kr_efde315e-f94f-4e8c-8335-9fd9f09074c6/metrics-server/0.log"
Jul 23 19:03:10.205090 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.205060    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-cfhkw_aeda98e7-9719-4a8d-9281-ef459dee0ca0/monitoring-plugin/0.log"
Jul 23 19:03:10.412756 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.412649    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z5q82_9227e00f-e0cf-4a30-b287-1c645cf59134/node-exporter/0.log"
Jul 23 19:03:10.438267 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.438134    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z5q82_9227e00f-e0cf-4a30-b287-1c645cf59134/kube-rbac-proxy/0.log"
Jul 23 19:03:10.463458 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.463302    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z5q82_9227e00f-e0cf-4a30-b287-1c645cf59134/init-textfile/0.log"
Jul 23 19:03:10.500585 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.500531    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-99qtp_005317cb-c24c-4cd2-ab7c-d3fc844b3e36/kube-rbac-proxy-main/0.log"
Jul 23 19:03:10.524756 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.524728    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-99qtp_005317cb-c24c-4cd2-ab7c-d3fc844b3e36/kube-rbac-proxy-self/0.log"
Jul 23 19:03:10.547703 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.547626    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-99qtp_005317cb-c24c-4cd2-ab7c-d3fc844b3e36/openshift-state-metrics/0.log"
Jul 23 19:03:10.587536 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.587505    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b428ca74-e113-4736-b602-6a4433562e84/prometheus/0.log"
Jul 23 19:03:10.613086 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.613056    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b428ca74-e113-4736-b602-6a4433562e84/config-reloader/0.log"
Jul 23 19:03:10.637734 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.637675    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b428ca74-e113-4736-b602-6a4433562e84/thanos-sidecar/0.log"
Jul 23 19:03:10.660330 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.660304    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b428ca74-e113-4736-b602-6a4433562e84/kube-rbac-proxy-web/0.log"
Jul 23 19:03:10.683213 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.683136    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b428ca74-e113-4736-b602-6a4433562e84/kube-rbac-proxy/0.log"
Jul 23 19:03:10.712262 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.712241    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b428ca74-e113-4736-b602-6a4433562e84/kube-rbac-proxy-thanos/0.log"
Jul 23 19:03:10.744304 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.744259    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b428ca74-e113-4736-b602-6a4433562e84/init-config-reloader/0.log"
Jul 23 19:03:10.773003 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.772976    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-lcxlm_bc0befc0-78e3-47f1-b522-826b27494639/prometheus-operator/0.log"
Jul 23 19:03:10.795202 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.795178    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-lcxlm_bc0befc0-78e3-47f1-b522-826b27494639/kube-rbac-proxy/0.log"
Jul 23 19:03:10.822064 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.822037    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-c9qkz_9e492398-f467-4e95-9138-3dc099fd9943/prometheus-operator-admission-webhook/0.log"
Jul 23 19:03:10.855694 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.855664    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77f855b89f-t6ps6_784ab2ff-0096-4a1d-958d-734cdc1ead62/telemeter-client/0.log"
Jul 23 19:03:10.898467 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.898422    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77f855b89f-t6ps6_784ab2ff-0096-4a1d-958d-734cdc1ead62/reload/0.log"
Jul 23 19:03:10.921001 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.920967    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77f855b89f-t6ps6_784ab2ff-0096-4a1d-958d-734cdc1ead62/kube-rbac-proxy/0.log"
Jul 23 19:03:10.951052 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.951023    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd98d6c6b-fpjv4_e6b89e84-d522-434f-b3c9-10131c183a9a/thanos-query/0.log"
Jul 23 19:03:10.974189 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.974155    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd98d6c6b-fpjv4_e6b89e84-d522-434f-b3c9-10131c183a9a/kube-rbac-proxy-web/0.log"
Jul 23 19:03:10.998850 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:10.998819    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd98d6c6b-fpjv4_e6b89e84-d522-434f-b3c9-10131c183a9a/kube-rbac-proxy/0.log"
Jul 23 19:03:11.023018 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:11.022989    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd98d6c6b-fpjv4_e6b89e84-d522-434f-b3c9-10131c183a9a/prom-label-proxy/0.log"
Jul 23 19:03:11.056989 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:11.056960    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd98d6c6b-fpjv4_e6b89e84-d522-434f-b3c9-10131c183a9a/kube-rbac-proxy-rules/0.log"
Jul 23 19:03:11.088419 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:11.088394    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cd98d6c6b-fpjv4_e6b89e84-d522-434f-b3c9-10131c183a9a/kube-rbac-proxy-metrics/0.log"
Jul 23 19:03:12.501703 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.501663    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"]
Jul 23 19:03:12.508636 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.508603    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.515711 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.515676    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"]
Jul 23 19:03:12.577511 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.577419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-proc\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.577827 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.577784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx656\" (UniqueName: \"kubernetes.io/projected/c024389e-daa8-4d15-a46e-b4d77fa73b30-kube-api-access-xx656\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.578030 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.577982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-sys\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.578268 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.578247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-lib-modules\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.578583 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.578568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-podres\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.679248 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.679209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-proc\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.679457 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.679271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx656\" (UniqueName: \"kubernetes.io/projected/c024389e-daa8-4d15-a46e-b4d77fa73b30-kube-api-access-xx656\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.679457 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.679314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-sys\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.679457 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.679361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-lib-modules\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.679457 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.679387    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-proc\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.679688 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.679472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-sys\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.679688 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.679525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-lib-modules\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.679688 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.679556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-podres\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.679830 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.679685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c024389e-daa8-4d15-a46e-b4d77fa73b30-podres\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.692307 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.688363    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx656\" (UniqueName: \"kubernetes.io/projected/c024389e-daa8-4d15-a46e-b4d77fa73b30-kube-api-access-xx656\") pod \"perf-node-gather-daemonset-9njkj\" (UID: \"c024389e-daa8-4d15-a46e-b4d77fa73b30\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:12.822514 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:12.822420    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:13.010664 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:13.009490    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"]
Jul 23 19:03:13.012323 ip-10-0-140-91 kubenswrapper[2561]: W0723 19:03:13.012230    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc024389e_daa8_4d15_a46e_b4d77fa73b30.slice/crio-6790bce0a3dfe998947ef8b39d4283189f8da35954165507c5dc3bd6334659c4 WatchSource:0}: Error finding container 6790bce0a3dfe998947ef8b39d4283189f8da35954165507c5dc3bd6334659c4: Status 404 returned error can't find the container with id 6790bce0a3dfe998947ef8b39d4283189f8da35954165507c5dc3bd6334659c4
Jul 23 19:03:13.514900 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:13.514860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj" event={"ID":"c024389e-daa8-4d15-a46e-b4d77fa73b30","Type":"ContainerStarted","Data":"5af15dfdd0e7d3c74580e80bbd558457fdb2855eb5ea14f0aefbb6008392ab70"}
Jul 23 19:03:13.515354 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:13.514922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj" event={"ID":"c024389e-daa8-4d15-a46e-b4d77fa73b30","Type":"ContainerStarted","Data":"6790bce0a3dfe998947ef8b39d4283189f8da35954165507c5dc3bd6334659c4"}
Jul 23 19:03:13.515354 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:13.514954    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:13.533622 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:13.533552    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj" podStartSLOduration=1.5335332670000001 podStartE2EDuration="1.533533267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:03:13.532900713 +0000 UTC m=+2789.380591241" watchObservedRunningTime="2026-07-23 19:03:13.533533267 +0000 UTC m=+2789.381223794"
Jul 23 19:03:15.196116 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:15.196083    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gg4kt_906c127e-72e7-4a3d-888d-7e0a5aa0cd9c/dns/0.log"
Jul 23 19:03:15.221599 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:15.221570    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gg4kt_906c127e-72e7-4a3d-888d-7e0a5aa0cd9c/kube-rbac-proxy/0.log"
Jul 23 19:03:15.385902 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:15.385876    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v4g5k_e804a5b4-b905-4966-a132-976a3d67e6dc/dns-node-resolver/0.log"
Jul 23 19:03:16.033387 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:16.033362    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rk2rs_40b7b2b4-947f-42bf-9f83-6bd57c73251e/node-ca/0.log"
Jul 23 19:03:17.099019 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:17.098986    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b75b7b5f6-598qm_2f75f153-7e86-4dc0-ad37-024ac81f3fba/kube-auth-proxy/0.log"
Jul 23 19:03:18.033343 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:18.033311    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b87z7_1a77fc19-bdd8-44c3-afc3-b03e0954adc7/serve-healthcheck-canary/0.log"
Jul 23 19:03:18.556149 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:18.556120    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7p9nd_ee48386e-15b8-49c1-b429-eaa71ba01cbd/kube-rbac-proxy/0.log"
Jul 23 19:03:18.578462 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:18.578433    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7p9nd_ee48386e-15b8-49c1-b429-eaa71ba01cbd/exporter/0.log"
Jul 23 19:03:18.598862 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:18.598831    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7p9nd_ee48386e-15b8-49c1-b429-eaa71ba01cbd/extractor/0.log"
Jul 23 19:03:19.535660 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:19.535620    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-9njkj"
Jul 23 19:03:21.390413 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:21.390303    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bdbd87db-hwlbr_37dedd14-286c-4b6e-a696-398b5b875b8e/manager/0.log"
Jul 23 19:03:21.406496 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:21.406463    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bdbd87db-hwlbr_37dedd14-286c-4b6e-a696-398b5b875b8e/copy-manifests/0.log"
Jul 23 19:03:22.830698 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:22.830612    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-778f8cb4df-jcp7l_ce843df6-358e-4733-a1b0-f7b9a0d0f716/manager/0.log"
Jul 23 19:03:22.857382 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:22.857347    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gtgq2_1c253c3b-da1b-4ec4-a684-24964a39afc1/openshift-lws-operator/0.log"
Jul 23 19:03:23.403027 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:23.403003    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b75b7b5f6-598qm_2f75f153-7e86-4dc0-ad37-024ac81f3fba/kube-auth-proxy/0.log"
Jul 23 19:03:24.831488 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:24.831419    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b75b7b5f6-598qm_2f75f153-7e86-4dc0-ad37-024ac81f3fba/kube-auth-proxy/0.log"
Jul 23 19:03:29.314856 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.314830    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d44gh_8a8557ac-27ae-4f77-8c7a-1d32266f570e/kube-multus-additional-cni-plugins/0.log"
Jul 23 19:03:29.336683 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.336651    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d44gh_8a8557ac-27ae-4f77-8c7a-1d32266f570e/egress-router-binary-copy/0.log"
Jul 23 19:03:29.358901 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.358875    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d44gh_8a8557ac-27ae-4f77-8c7a-1d32266f570e/cni-plugins/0.log"
Jul 23 19:03:29.380236 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.380212    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d44gh_8a8557ac-27ae-4f77-8c7a-1d32266f570e/bond-cni-plugin/0.log"
Jul 23 19:03:29.402641 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.402618    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d44gh_8a8557ac-27ae-4f77-8c7a-1d32266f570e/routeoverride-cni/0.log"
Jul 23 19:03:29.421070 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.421050    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d44gh_8a8557ac-27ae-4f77-8c7a-1d32266f570e/whereabouts-cni-bincopy/0.log"
Jul 23 19:03:29.440451 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.440431    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d44gh_8a8557ac-27ae-4f77-8c7a-1d32266f570e/whereabouts-cni/0.log"
Jul 23 19:03:29.632076 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.632011    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h86v2_abc48b6f-98c5-43b6-8a1e-edd9d21e0ccb/kube-multus/0.log"
Jul 23 19:03:29.716064 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.716038    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-75jnc_4e9413cd-4d10-4609-a427-55fcc8474d38/network-metrics-daemon/0.log"
Jul 23 19:03:29.735235 ip-10-0-140-91 kubenswrapper[2561]: I0723 19:03:29.735202    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-75jnc_4e9413cd-4d10-4609-a427-55fcc8474d38/kube-rbac-proxy/0.log"